#421 committed the compiled tests/test_st_pread (mode 644, non-executable).
On a fresh CI checkout make sees it as up-to-date and skips rebuilding, so
run_tests.py hits 'Permission denied' exec'ing a non-executable file — the
linux/macos 'make check' failure. Remove the binary and gitignore it next to
the existing test_st_mirror rule.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per-request grammar constraints in serve mode: response_format plumbed through the API, per-request grammar compilation, and grammar-forced draft verification. Verified locally on top of current dev: clean build, token-exact tiny models unchanged, decode-batch helper tests ok, 40/40 openai_server tests. CI 8/8 green. Thanks @fabio-rovai for keeping this rebased through the refactor.
Tools-only step of the #452 plan: iq3_pack.py index codec + IQ3_XXS grid + quant_ablation hook, with round-trip tests (6/6 locally, CI 8/8). No engine changes.
Co-authored-by: ZacharyZcR
CI keeps the runtime path dependency-free, so the Python test job runs
without numpy and the new codec tests errored on import. Guard the import
and skip the class instead — verified both ways: 6/6 run where numpy is
present, 6/6 skip cleanly where it is not.
Dev split coli_cuda_tensor_upload into two symbols: the plain 7-arg
tensor_upload (no gs) for fmt!=4, and tensor_upload_g (8-arg, with gs)
for fmt=4 grouped. The DLL-side _g sets a g_upload_gs global the plain
upload reads internally. Adapted all call sites:
- colibri.c: qt_cuda_upload already dispatches (_g for fmt=4, plain
otherwise) — kept dev's version. layer_cuda_shard_kvb was calling the
plain upload with gs; switched to tensor_upload_g.
- backend_loader.c: the C-API wrapper matches dev's 7-arg signature;
tensor_upload_g dispatched separately.
- backend_cuda.cu: cache-hit check uses g_upload_gs (not a gs param);
coli_cuda_matmul dispatches _g when gs>0; the grouped-expert host
fallback quant_matmul calls pass gs=0,ng=1 (host tier is per-row).
- Kept dev's fault-injection hook, scale_count field, and fmt=4 support
in the grouped-expert path (all_q4/any_g4 tracking).
- tests: kept dev's comprehensive tensor_upload test (cached reuse,
temp-buffer survival, upload-failure accounting, fault injection).
Build-verified: colibri.exe + coli_cuda.dll both compile clean.
Dev refactored glm.c → colibri.c and extracted the matmul/quant kernels
into quant.h (matmul, matmul_q, matmul_i4, matmul_i4_grouped, matmul_i2,
quant_scratch, dot_i4i8, matmul_q_idot, matmul_i4_idot, etc. all live
there now). The original #298 commit re-added all of these inline; on
rebase they became duplicate definitions.
Resolution:
- Removed the ~700-line duplicate block (everything dev moved to quant.h)
- Kept ONLY the unique fmt=4 contribution: matmul_i4_grouped_pair (the
fused gate+up kernel that reads x once instead of twice, ~33% decode
speedup) + the fmt=4 branch in expert_gate_up that dispatches to it.
Dev's expert_gate_up only fused fmt==2; this adds the fmt==4 case.
- Forward-declared matmul_i4_grouped_pair before expert_gate_up.
- Fixed quant_matmul call site in the ragged attention path (backend_cuda.cu)
to pass gs/ng — the kernel signature gained those args in the attention
scales fix, but dev's new ragged path called it with the old signature.
Build-verified: colibri.exe (CPU + COLI_CUDA) and coli_cuda.dll both
compile clean on the rebased branch.
PR #298 added fmt=4 (grouped int4, gs=64) support to quant_matmul but the
two MLA attention absorb kernels kept the per-row scale semantic
(wscale[row]) from fmt=2. The g64 model's kv_b is fmt=4 (ng=8 groups/row),
so COLI_CUDA_ATTN=1 / COLI_CUDA_PIPE=2 routed it through attention_absorb*
which indexed the O*ng scale array with a row index -> wrong stride ->
GPU memory fault -> bugcheck 0x116 VIDEO_TDR_FAILURE -> reboot.
Add absorb_scale() (mirrors quant_matmul's fmt==4 branch: wscale[row*ng+k/gs]
for fmt=4, wscale[row] otherwise) and apply it inside the Q- and V-projection
accumulation loops of both attention_absorb_kernel and attention_absorb_batch_kernel.
Thread w->gs/w->ng through all six launch sites. No extern-C signature or
header changes; the tensor already carries gs/ng from tensor_upload. For
fmt!=4 ng==1 so k/gs==0 and the result is bit-identical to before.
Validated: COLI_CUDA_ATTN=1 and COLI_CUDA_PIPE=2 (long-prompt prefill, the
exact crash config) now run clean; base path unchanged.
The kv_b shard scale pointer used h0*(Q+V) which is correct for per-row
scales (fmt=2: one scale per row). For fmt=4 (grouped), there are ng
scales per row, so the offset must be h0*(Q+V)*ng. Without this, the
shard reads from the wrong scale position on multi-GPU, producing silent
corruption. Single-GPU is unaffected (no sharding).
Fix: const float *scale=l->kv_b.s+(int64_t)h0*(Q+V)*(gs>0?ng:1);
Refs #298
The new g64 model quantizes experts with group-size 64 (fmt=4): one f32
scale per 64 elements instead of per-row. This required changes across
the entire compute stack — CUDA kernel, engine dispatch, and dequant
helpers — plus a new fused gate+up kernel to recover the ~40% throughput
that the generic grouped path costs.
CUDA backend (backend_cuda.cu):
- ColiCudaTensor: add gs/ng fields for group-size metadata
- row_bytes/weight_at: fmt=4 uses same packed int4 layout as fmt=2
- quant_matmul: apply per-group scales (scales[o*ng+g]) for fmt=4,
matching the CPU matmul_i4_grouped accumulation exactly
- coli_cuda_tensor_upload: allocate O*ng scales (not O) for fmt=4,
store gs/ng on the tensor
- coli_cuda_tensor_update: handle grouped scales on refresh
- coli_cuda_tensor_free/tensor_bytes: VRAM accounting uses O*ng
- coli_cuda_expert_group: return 0 for fmt=4 (fall back to correct
per-expert path, since grouped_hidden/grouped_down kernels only
handle per-row scales)
- All 11 quant_matmul call sites updated to pass gs,ng
Engine (glm.c):
- qt_cuda_upload: pass t->gs to tensor_upload
- matmul dispatch (line 816): pass w->gs to coli_cuda_matmul
- kv_b shard upload: pass l->kv_b.gs
- kv_b shard rb computation: add fmt=4 case ((I+1)/2)
- embed_row: add fmt=4 branch with per-group scale dequant
- qt_addrow/qt_matvec_rows: add fmt=4 branches (CPU MLA absorption path)
- expert_gate_up: add matmul_i4_grouped_pair — fused gate+up for fmt=4
that reads x once instead of twice (+40% tok/s, 0.77 -> 1.08)
- run_text: prepend [gMASK]<sop> prefix for GLM models (#108 fix —
without it, PROMPT mode generates garbage on all GLM snapshots)
API (backend_cuda.h, backend_loader.c):
- coli_cuda_tensor_upload and coli_cuda_matmul signatures: add gs param
- Loader typedefs and wrappers thread gs through the DLL boundary
Tests:
- bench_tensor_core.cu, test_backend_cuda.cu, test_pipe_cuda.cu:
update all calls to new API signature (append gs=0 for non-grouped)
New tool: c/tools/diag_harness.py
- Comprehensive model diagnostic harness (system probe, correctness
smoke, deep PROFILE diagnostic, quality benchmarks via eval_glm.py,
throughput with/without MTP). Outputs JSON + Markdown reports.
Performance (GLM-5.2 744B g64 / RTX 5070 Ti / 32GB RAM):
broken CUDA: 0.05 tok/s (every tensor fell back to CPU)
fixed CUDA: 0.30 tok/s (6x — CUDA working)
+ full opt stack: 0.77 tok/s (CACHE_ROUTE + EXPERT_BUDGET)
+ fused grouped pair: 1.08 tok/s (+40% from gate+up fusion)
The old subprocess.run(capture_output=True) buffered every score result in
memory until the engine exited, then parsed+scored them all at once. If the
engine crashed (or was killed) at request 39/40 you lost everything -- a
multi-hour run with zero information, which is exactly what happened on the
g64 eval against a disk-I/O-bound 400GB model.
Now streams engine stdout line-by-line via Popen:
- each score result lands in the CSV immediately (flushed), with full
provenance (task/qi/oi/gold/logprob/greedy + a header with snap/tasks/
limit/seed/timestamp)
- a [progress] line every 5 requests: N/total, elapsed, req/s, ETA, last
request scored
- the engine's own [score N req] stderr heartbeat streams live too
- on partial completion (crash/kill at request N): keeps 1..N-1, fills the
rest with -inf, and scores the partial results -- never a wasted run
New --out <path> writes the incremental CSV; without it, behaves as before
(results to stdout only). Backward compatible: all existing args unchanged.
Routed experts (gate/up/down) can now take different bit widths via PROJ_BITS.
Motivating config: --xbits 4 --up-bits 3 puts up_proj at int3-g64 (fmt=5, this
PR's format) while gate/down stay int4 — ~8% fewer expert bytes on disk and per
token, at ~zero quality cost. Backed by the OLMoE per-projection ablation posted
to #168: up@int3 matches int4-g64 (56.2 vs 55.8), up@int2 craters (-16pp).
This also supplies the definition the #404 resume manifests already depend on:
current dev records dict(PROJ_BITS) in check_or_record_params and the --indir
progress file in four places, but the global was never defined — every one of
those paths NameErrors at runtime today. The manifests were written for this
interface; this commit is the other half.
Validated: synthetic GLM fixture with --up-bits 3 yields int3-g64 up_proj
(O*(I/64)*24B weight + group scales) and int4-per-row gate/down.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New weight format: int3 with ONE f32 scale per 64-input group (3.5 bits/weight
effective). Per group: 16B low plane (2 bits/val, int2 layout) + 8B high plane
(1 bit/val), values [-4,3] stored v+4. Same quantization math as
tools/quant_ablation.py _quant_last_dim(bits=3, group=64) from #132, whose
OLMoE ablation measured int3-g64 BEATING the shipped per-row int4 on quality
(-7.5 vs -9.3pp) at ~14% fewer bits.
Engine (placed per the #391 split): matmul_i3 + pack_int3_g64 + I3_* layout
helpers in quant.h next to their kernel family; fmt=5 branches in colibri.c's
qt_bytes/qt_alloc/qt_fill/matmul_qt/embed_row/qt_addrow/qt_matvec_rows.
Format detection now goes through #413's qt_resolve_fmt: fmt=5 registers its
distinct weight-byte layout O*ceil(I/64)*24 and its scale cardinality
O*ceil(I/64) there, validated against [O,I] like every other format. int3-g64
and grouped-int4-at-gs=64 carry the SAME scale count, so the weight bytes are
the int3 tag; row formats keep precedence for the small-I shapes where byte
counts coincide. The io_uring expert path still used the raw ?1:?2:3 byte
inference (it missed fmt=4 grouping entirely and never set gs) — converted to
qt_resolve_fmt like the other two expert paths.
Backends: qt_cuda_upload returns 0 for fmt=5 (tensor stays CPU-side, the
documented fallback), the dense CUDA matmul gate excludes fmt=5, and Metal's
existing fmt gates (gemm fmt<=3, moe fmt 1/2) already reject it.
Converter: quant_int3_g64 in convert_fp8_to_int4.py; --ebits 3/--xbits 3 now
emit it (previously bits=3 silently produced int4).
Tests: tests/test_int3.c (bit-exact pack/unpack vs reference, matmul_i3 vs
dequant-matmul incl. short tail groups and the real GLM I=7168, QT plumbing,
qt_resolve_fmt disambiguation incl. the same-scale-count fmt=4/fmt=5 pair,
outlier-rows RMS: int3-g64 3.3x lower error than per-row int4),
tests/test_int3_load.c (hand-rolled .safetensors fixture through st_init +
qt_from_disk: fmt=5 detected and loaded next to an int4 control tensor),
tests/test_int3_convert.py (NumPy pack round-trip vs independent decoder).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Addresses the #192 review: server usage documented in docs/grammar-draft.md
(incl. back-compat statement for the additive SUBMIT field and the #100-class
near-tie caveat); gateway pre-checks grammar payloads at 1 MiB (matching the
engine's gbytes bound); negative tests for non-dict response_format, empty and
oversized grammars, plus an explicit test that malformed GBNF passes the
gateway by design (engine fail-soft, draft-source semantics). Measured compile
overhead: 7.8 us/request typical schema, 17.9 us at the 32-level nesting cap.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The OpenAI gateway previously 400'd every response_format; the mux engine path
ran with speculation disabled entirely. Now:
- openai_server.py: response_format {"type":"json_object"} (generic ws-tolerant
JSON grammar), {"type":"json_schema"} (schema forwarded as-is, compiled
engine-side by schema_gbnf.h), and a raw-GBNF {"type":"gbnf"} extension.
Draft-source semantics throughout: a schema the engine cannot compile costs
the speedup, never the request and never the output.
- SUBMIT protocol: optional 7th field gbytes; grammar text appended to the
payload after the prompt. 6-field headers unchanged (back-compatible).
- Engine: per-slot GrDraft (grammar_setup_text/grammar_teardown split out of
the env-driven setup); walkers fed on every emitted token. Grammar-forced
drafting in run_serve_mux for greedy requests: a drafting slot leaves the
shared batch for one forward and runs the proven single-sequence verify path
(kv_bind + step_all) — the same primitives prefill already uses per
submission — then rejoins; rejected drafts' KV entries are overwritten by the
next forward exactly like the existing prefix-truncation path. Sampling
requests never draft (verification under sampling needs rejection resampling;
out of scope).
Tests: 7-field SUBMIT parse cases; response_format->grammar plumbing incl.
fail cases; test doubles updated; generic JSON grammar parse+walk validated
against grammar.h. make test-c green; python suite green except the known
environmental memory_available failure (#150 fixes it).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The #270 rebase resolved the TEST_BINS list but missed the test_pipe_block
rule prerequisite and its #include, both still pointing at glm.c (which #391
renamed to colibri.c) — 'No rule to make target glm.c' broke the Linux C test
suite on dev. Point both at colibri.c. Build-verified locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
On Windows a bare 'coli chat' (no --gpu/--vram/--auto-tier) ALWAYS ran
CPU-only, even on a CUDA build with a GPU present. Two defects:
1. cuda_binary() returned False on Windows. It detects CUDA by running
'ldd glm | grep libcudart', which is Linux-only (no ldd on win32) and
meaningless anyway because the Windows engine links cudart only inside a
runtime-loaded coli_cuda.dll, not as a libcudart symbol in glm.exe. So
the --gpu/--vram/--auto-tier gates (which call cuda_binary()) never opened.
2. Even with detection fixed, bare 'coli chat' set no CUDA env: env_for's
else-branch only enables CUDA when --gpu/--vram is passed. Nothing
auto-enabled the GPU.
Now: cuda_binary() on non-Linux returns True iff coli_cuda.dll exists next
to glm.exe — the exact file backend_loader.c loads from the engine's own
directory, so its presence is a faithful, cheap, DLL-hijack-safe proxy for a
CUDA-capable build. And env_for, scoped to win32 (Linux keeps its working
explicit-flag UX), auto-enables CUDA when a bare chat detects a CUDA build
plus a GPU via nvidia-smi, sizing the expert-tier VRAM budget from real free
VRAM via the existing build_plan/environment_for_plan machinery (same as
--auto-tier, no guessed budget). If nvidia-smi is missing it falls back to
CPU with a clear warning; --gpu none still forces CPU; explicit --vram/--gpu
still win. CUDA_DENSE stays an explicit opt-in (matches --auto-tier).
Verified on a Windows + RTX 5070 Ti box: bare 'coli chat --model <g64>' now
prints '[GPU] auto-enabled CUDA ... 13.0 GB expert tier' and emits
COLI_CUDA=1 / COLI_GPUS=0 / CUDA_EXPERT_GB=13.044 (was: all unset, CPU-only).
Tests: 4 new cases (auto-enable, nvidia-smi-missing fallback, CPU-build
silent, Linux-unchanged) plus the 4 existing default-I/O tests guarded to
mock cuda_binary() so they stay host-independent. Full python suite green
(env_defaults 8, resource_plan 10, doctor 8, makefile_platform 3, cli_output 3).
Out of scope: doctor.cuda_linkage is also POSIX-only and mis-reports on
Windows — separate follow-up.
JustVugg caught a regression by running the PR: the code asked
`lscpu -p=CPU,Core,Socket` and read fields[1]/fields[2], but the comment's
claim was inverted -- `lscpu -p=<list>` emits EXACTLY the requested columns
(no CPU prefix), while bare `lscpu -p` prepends CPU.
On machines where the requested list is short or lscpu collapses to two
columns, every line was skipped by the `< 3` guard, cores stayed empty, and
the code fell through to os.cpu_count() -- the LOGICAL count. Result: 6
physical cores reported as 12 (SMT over-subscription), the opposite of the
fix.
Correct per review:
- ask lscpu for exactly 'core,socket'
- take fields[-2], fields[-1] (correct whether or not CPU is prepended)
- keep the warning scaffolding + the _resolve_physical_cores clamp
(those are the actual #325 fix; only the indexing was wrong)
Test now exercises BOTH the 2-column (-p=core,socket) and 3-column
(bare -p, CPU prefix) layouts, asserting 12 physical cores each. The
2-column case is the one that regressed: old parser returns 24, new
returns 12.
The core-count fix was necessary but not sufficient: @liangstein confirmed
physical_cpu_count() now returns 64 on his box, yet --auto-tier STILL pinned
decode to one core. A complete env diff between the plain (working) and
--auto-tier (broken) paths showed exactly three keys the plan adds:
OMP_NUM_THREADS = 64 (correct, verified)
OMP_PROC_BIND = spread
OMP_PLACES = cores
Since OMP_NUM_THREADS was already correct, the culprit is the affinity pair.
The mechanism: environment_for_plan() sets OMP_PROC_BIND=spread + OMP_PLACES=cores
in the launcher's env. The engine's hot-thread tuning (glm.c main, the
COLI_OMP_TUNED self-exec) then tries setenv("OMP_PROC_BIND","close", overwrite=0)
-- but overwrite=0 cannot replace an already-set var, so the plan's "spread"
wins. On the reporter's libgomp + multi-socket topology, spread + places=cores
collapsed the team to a single CPU even with 64 threads configured.
Fix: don't set affinity from the plan at all. The engine deliberately chose
"close" for cache locality (the tiny back-to-back per-expert matmuls want
adjacent cores), and the plain path already leaves affinity to the engine.
Removing the plan's spread/places makes --auto-tier match the working plain
path; a user wanting a specific policy can still set OMP_PROC_BIND/OMP_PLACES
in their own environment (environment_for_plan only setdefaults OMP_NUM_THREADS).
Verified via env diff: after the fix, --auto-tier adds ONLY OMP_NUM_THREADS
beyond the plain env -- the engine's own close/bind tuning now wins on both
paths identically.
Note: this could not be reproduced on Windows (MinGW libgomp prints "Affinity
not supported on this configuration" and ignores the vars entirely); it is
Linux-libgomp-specific, matching the reporter's Rocky 9 box.
Tests: rewrite test_applies_plan_without_overriding_explicit_settings to assert
the plan sets NO affinity vars on any platform (the old test encoded the buggy
platform-dependent spread/cores contract). Add
test_plan_does_not_set_omp_affinity_vars as a focused regression. 78/78 pass.
physical_cpu_count() silently returned 1 in two situations, and that value
flowed through build_plan -> OMP_NUM_THREADS to pin every matmul region to a
single thread under --auto-tier (reported on Rocky Linux 9, 512 GB RAM).
Two root causes:
1. The lscpu parse counted the wrong thing. `lscpu -p=core,socket` prepends the
CPU column, so the output is actually CPU,Core,Socket; the old set
comprehension collected (CPU,core,socket) tuples that were unique per logical
CPU. Now parse CPU,Core,Socket and dedupe on (core, socket) to get true
physical cores (the SMT-doubling the surrounding comments warn against was
the actual behavior).
2. Any probe failure fell through to `os.cpu_count() or 1`. On a cgroup'd or
otherwise constrained box os.cpu_count() can be 1 (or None), silently
capping the run. Skip offline core/socket fields ("-" instead of raising
ValueError) so a single offline row no longer discards the whole parse, and
replace the silent `or 1` fallback with os.cpu_count() (logical) plus an
explicit warning. Only return 1 when nothing at all is detected, and warn.
Also harden the win32 branch: declare argtypes/restype on
GetLogicalProcessorInformationEx (an undeclared 64-bit WinAPI returns c_int and
takes c_int pointers, so the probe could silently fail), and warn on its
fallbacks. Replace the silent max(1, int()) clamp in build_plan with
_resolve_physical_cores() that clamps to 1 with a warning instead of masking.
Tests: the existing OMP_NUM_THREADS test passed physical_cpus=24 explicitly, so
it never exercised the real probe -- that is why this regressed. Add regression
coverage for the lscpu physical-core dedup, offline fields, lscpu-missing
fallback, zero-cores degenerate case, and an end-to-end build_plan +
environment_for_plan check that OMP_NUM_THREADS reflects physical (not logical)
cores.
CI runs 'make check' = dependency-free tests, no model downloads (by design,
#140). glm_tiny/ is a gitignored generated fixture, so test_inefficiency.py
hard-failed on the Windows/macOS/Linux runners with 'config.json: No such file
or directory' instead of skipping.
_engine_present() now requires BOTH glm.exe AND glm_tiny/config.json, and
_skip_reason() names exactly which prerequisite is missing so the skip is
actionable. Verified: 8 skipped (0 failed) with the fixture absent; 5 pass +
3 CUDA-skip with it present.
Two layers of efficiency coverage for the engine, both parsing the telemetry
glm.c already emits (REPLAY/PROFILE/[PROF]/CUDA-tier) but nothing previously
asserted on:
1. test_inefficiency.py — tiny-model asserted regression tests (8 tests, run
in make test via test-python). Gate on: throughput floor, PROFILE phase
accounting sanity, disk-wait not dominant on a resident model, CPU greedy
determinism, and (when a CUDA build is present) CUDA init, dense VRAM
upload, and CPU-vs-CUDA argmax agreement >= 70%. CUDA tests auto-skip with
a clear build hint on CPU-only binaries.
2. test_efficiency_report.py — opt-in optimization dossier for a real model.
Turns on every instrumentation flag (PROF, COLI_CUDA_PROFILE, CACHE_ROUTE,
DISK_SPLIT, LOOKA) and prints 9 sections (provenance, throughput + tail
latency, where-time-goes, attention breakdown, expert cache, disk I/O +
phase split, routing quality + predictability, speculation, GPU tiers),
each flagging inefficiency with the concrete knob to move tok/s. Never
fails CI.
tools/efficiency.py is the shared harness: parse_run() captures every signal,
run_engine() wraps the subprocess. Reuses PROFILE_RE/SPEED_RE from
tools/benchmark_cuda_fixture.py and extends the tok/s regex to also catch the
run_text (parenthesized) format the full-model PROMPT path uses.
Makefile adds: efficiency / efficiency-cuda / efficiency-report targets.
Verified end-to-end on the full glm52_i4_g64 model (CPU + CUDA).
#453 settled the scheme; this produces the actual bytes. iq3_pack.encode/decode
implement the container layout so the converter, the engine and the decode
kernels can all be written against one spec:
98 bytes per 256 weights = 3.0625 bpw
[ 0..63] uint8 grid index per 4-dim magnitude block
[64..95] uint32 x8 — four 7-bit sign words + 4-bit sub-scale per 32
[96..97] fp16 super-scale
Signs use the published odd-parity trick: 7 of every 8 are stored and the 8th
is derived, so the encoder flips the smallest-magnitude weight of any block
whose true signs would violate parity — the cost the #453 ablation priced in,
now actually paid. Sub-scale is searched over all 16 codes against the stored
(fp16-rounded) super-scale, so encode-time and decode-time arithmetic agree.
GLM-5.2 routed experts under this container: 372.7 -> 281.2 GB (-24.6%), and a
176 GB VRAM tier holds ~12,180 experts instead of ~9,190 (+33%).
tests/test_iq3_pack.py: byte budget, deterministic encode, decode checked
value-by-value against an independent loop-based reader written straight from
the layout, sign-parity closure, and reconstruction quality in the band the
chosen scheme measured (rel-RMSE 0.195 vs the torch model's 0.195).
Upstream courtesy fix, found while auditing the conversion recipe for this
branch -- pre-existing in dev, not introduced by int3-g64, but directly
relevant to anyone converting for real (a #383-class gap: same failure
family as the resume/manifest work already done for --indir, and the
silent-mixing mechanism issue #355 fixed for a narrower case).
The --indir path already refuses to resume with different conversion
parameters on the same --outdir (a manifest records ebits/xbits/io_bits/
group_size/n_layers/bits_map and compares on every resume). The --repo
streaming download loops (main model, --mtp, --indexer) never got the
same guard: each shard's resume check is just `if os.path.exists(outp):
continue` -- true whether or not THIS run's flags match the flags that
produced that shard. A --repo conversion resumed with changed bits
(--xbits 3 -> 4 mid-run, say, after an interruption) would silently mix
bit-widths across shards in the same container, with no error and no log
line distinguishing it from a normal resume.
Fix: check_or_record_params(), a small shared helper mirroring the
--indir manifest's refuse-on-mismatch logic but without needing its
per-shard bookkeeping (the --repo loops already track shard completion
correctly via out-NNNNN.safetensors existence, since shard index maps
directly to output filename there -- only whether the params used SO FAR
still match needed adding). Applied to all three --repo loops with
per-mode sidecar files (.out-mtp-params.json / .out-idx-params.json /
.out-params.json) so a --mtp and a main-model conversion into the same
--outdir don't cross-check each other's parameters.
Also includes PROJ_BITS (the per-projection expert bit overrides) in the
tracked params dict on BOTH paths -- it was missing from --indir's
existing manifest too, so a resume with a changed --up-bits/--gate-bits/
--down-bits would have passed the existing guard silently.
Verified directly (no real HF downloads; --repo network paths can't be
exercised under this task's constraints): unit-tested
check_or_record_params() standalone -- fresh outdir accepts and records,
a same-params resume accepts, a changed --xbits is refused, and a
proj_bits-only change (nothing else different) is refused. Re-ran the
existing --indir dry-run end to end (convert, resume, resume-with-changed-
xbits) to confirm the manifest-based path still works correctly with
proj_bits added to its params dict.
Gates: make test-c (20/20) and make test-python (85/85) both pass.
Adds `-iq3` to the ablation harness: a faithful torch model of llama.cpp's
deployed 3.06-bpw IQ3_XXS format — 256-entry 4-dim magnitude grid
(extracted from ggml-common.h, MIT), signs factored per 8 weights with
the odd-parity constraint priced in (a violating block flips its
smallest-magnitude sign), fp16 super-scale per 256 + 4-bit sub-scale per
32 searched over all 16 codes. Nearest-grid search runs as chunked
matmul-argmin (|g|^2 - 2 q.g) — a full cdist materializes tens of GB on
a 100M-param tensor and OOMed the first run.
Measured (OLMoE-1B-7B, n=200 x hellaswag/arc/mmlu; the first four rows
reproduce the published ablations exactly):
fp16 58.0%
int4 per-row 48.7% (-9.3pp, the shipped container's scheme)
int3-g64 50.5% (-7.5pp)
int3-g64-e8-rot 51.5% (-6.5pp, simulated rate-scaled ball)
int3-iq3 49.3% (-8.7pp)
int3-iq3-rot 51.5% (-6.5pp)
The deployable IQ3 codebook plus rotation exactly ties the simulated E8
ball — that settles #452's codebook decision toward the IQ3-style block
structure, with rotation mandatory (worth 2.2pp on this codebook).
The VRAM-ranked prefix's host slabs are upload staging — read once and
freed right after — so binding them buys nothing and cost ~2 transient
VMAs each: measured PEAK maps 28,958 on the six-GPU host even with the
pin arenas in place. With the skip: 11,771, all of it the bounded LRU
ecache, which serves decode reads and stays correctly bound.
Root cause of #419's 'OOM slab': every per-slab mbind carries its own
memory policy, so bound regions cannot merge — measured ~2 VMAs per slab,
with or without MPOL_MF_MOVE. A PIN_GB=all load (19,456 experts x
slab+fslab) creates ~78k VMAs and crosses the default
vm.max_map_count=65530: posix_memalign dies with terabytes free.
The fix binds the pinned hot-store as ONE arena per layer. Experts of a
layer share a tensor shape, so a layer's pins pack at a fixed stride into
two arenas (weights + scales): 2 mbinds and a handful of VMAs per layer
instead of ~500. Slices are pre-attached to the slots before the load
loop — slab_cap covers expert_load's realloc check, so its alloc branch
never fires and expert_load itself is untouched. aslab marks arena
ownership: expert_host_release detaches instead of freeing (a REPIN
gpu-swap promotion must not free() an interior pointer), and
expert_host_ensure re-attaches the slice before reloading.
Per-slab mbind remains for the bounded allocations (dense qalloc, LRU
ecache, GPU-tier staging), now without MPOL_MF_MOVE: every bind lands
before the pread that first-touches the pages, so there is nothing to
migrate.
numa_init also gains a capability probe done right: one page-aligned
page (mbind rejects unaligned addresses with EINVAL), disabling only on
errno==EPERM — so a constrained container degrades with a message
instead of crashing later, and an EINVAL can never masquerade as a
missing capability.
The arena path activates only when interleave is actually on
(g_numa_nodes>=2, Linux, non-mmap): default builds stay byte-identical.
The grouped MoE kernels were per-row-only: GroupDesc had no group-size
fields, row_bytes() returned 0 for fmt=4, the scale buffer was hardcoded
to O floats, and a fmt=4 group that reached the generic path would have
been silently decoded as int2. This closed the GPU expert tier to every
grouped container — including the g64 quality line (#225) and the E8
lattice route (#347) whose whole point is fitting more experts in VRAM.
- ColiCudaTensor gains gs/scale_count; upload allocates O*ceil(I/gs)
scales for fmt=4 and applies the same offset->signed nibble conversion
as fmt=2 (identical packing). New ABI entry coli_cuda_tensor_upload_g
carries gs without touching the existing symbol — an old Windows DLL
missing it returns 0 and the tensor simply stays CPU-side.
- GroupDesc gains per-tensor group sizes; new grouped_hidden_g4_dual /
grouped_down_g4 apply the per-group scale inside the accumulation
(gs is required even, so a packed byte never straddles groups; gs=0
degrades to per-row, letting fmt=2 members ride the same launch).
- coli_cuda_expert_group routes any group containing fmt=4 through the
g4 kernels; pure-fmt=2 groups keep the existing paths byte-identical.
The generic fallback now explicitly rejects fmt=4 instead of decoding
garbage (#334's prevention note, made real).
tests/test_grouped_g4_cuda.cu: kernel-vs-CPU oracle over 50 trials x 3
experts — gs=64, a non-divisible tail group (200 % 64), and a per-row
member in the same launch: zero mismatches on a 5090.
make check 77/77; CPU, CUDA and MinGW builds clean.
JustVugg