New weight format: int3 with ONE f32 scale per 64-input group (3.5 bits/weight
effective). Per group: 16B low plane (2 bits/val, int2 layout) + 8B high plane
(1 bit/val), values [-4,3] stored v+4. Same quantization math as
tools/quant_ablation.py _quant_last_dim(bits=3, group=64) from #132, whose
OLMoE ablation measured int3-g64 BEATING the shipped per-row int4 on quality
(-7.5 vs -9.3pp) at ~14% fewer bits.
Engine (placed per the #391 split): matmul_i3 + pack_int3_g64 + I3_* layout
helpers in quant.h next to their kernel family; fmt=5 branches in colibri.c's
qt_bytes/qt_alloc/qt_fill/matmul_qt/embed_row/qt_addrow/qt_matvec_rows.
Format detection now goes through #413's qt_resolve_fmt: fmt=5 registers its
distinct weight-byte layout O*ceil(I/64)*24 and its scale cardinality
O*ceil(I/64) there, validated against [O,I] like every other format. int3-g64
and grouped-int4-at-gs=64 carry the SAME scale count, so the weight bytes are
the int3 tag; row formats keep precedence for the small-I shapes where byte
counts coincide. The io_uring expert path still used the raw ?1:?2:3 byte
inference (it missed fmt=4 grouping entirely and never set gs) — converted to
qt_resolve_fmt like the other two expert paths.
Backends: qt_cuda_upload returns 0 for fmt=5 (tensor stays CPU-side, the
documented fallback), the dense CUDA matmul gate excludes fmt=5, and Metal's
existing fmt gates (gemm fmt<=3, moe fmt 1/2) already reject it.
Converter: quant_int3_g64 in convert_fp8_to_int4.py; --ebits 3/--xbits 3 now
emit it (previously bits=3 silently produced int4).
Tests: tests/test_int3.c (bit-exact pack/unpack vs reference, matmul_i3 vs
dequant-matmul incl. short tail groups and the real GLM I=7168, QT plumbing,
qt_resolve_fmt disambiguation incl. the same-scale-count fmt=4/fmt=5 pair,
outlier-rows RMS: int3-g64 3.3x lower error than per-row int4),
tests/test_int3_load.c (hand-rolled .safetensors fixture through st_init +
qt_from_disk: fmt=5 detected and loaded next to an int4 control tensor),
tests/test_int3_convert.py (NumPy pack round-trip vs independent decoder).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The #270 rebase resolved the TEST_BINS list but missed the test_pipe_block
rule prerequisite and its #include, both still pointing at glm.c (which #391
renamed to colibri.c) — 'No rule to make target glm.c' broke the Linux C test
suite on dev. Point both at colibri.c. Build-verified locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The release ships colibri-<ver>-windows-x86_64.zip but nothing said what the
.exe is for or how to use it. The quickstart's Windows Option A now lists the
zip contents (engine .exe / coli launcher / Python support), and gives the two
missing steps: rename the engine to glm.exe so the coli launcher finds it, and
install Python 3 for the launcher/gateway. README's run section gets a short
Windows-prebuilt pointer to the same. Docs-only.
Closes#450
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
On Windows a bare 'coli chat' (no --gpu/--vram/--auto-tier) ALWAYS ran
CPU-only, even on a CUDA build with a GPU present. Two defects:
1. cuda_binary() returned False on Windows. It detects CUDA by running
'ldd glm | grep libcudart', which is Linux-only (no ldd on win32) and
meaningless anyway because the Windows engine links cudart only inside a
runtime-loaded coli_cuda.dll, not as a libcudart symbol in glm.exe. So
the --gpu/--vram/--auto-tier gates (which call cuda_binary()) never opened.
2. Even with detection fixed, bare 'coli chat' set no CUDA env: env_for's
else-branch only enables CUDA when --gpu/--vram is passed. Nothing
auto-enabled the GPU.
Now: cuda_binary() on non-Linux returns True iff coli_cuda.dll exists next
to glm.exe — the exact file backend_loader.c loads from the engine's own
directory, so its presence is a faithful, cheap, DLL-hijack-safe proxy for a
CUDA-capable build. And env_for, scoped to win32 (Linux keeps its working
explicit-flag UX), auto-enables CUDA when a bare chat detects a CUDA build
plus a GPU via nvidia-smi, sizing the expert-tier VRAM budget from real free
VRAM via the existing build_plan/environment_for_plan machinery (same as
--auto-tier, no guessed budget). If nvidia-smi is missing it falls back to
CPU with a clear warning; --gpu none still forces CPU; explicit --vram/--gpu
still win. CUDA_DENSE stays an explicit opt-in (matches --auto-tier).
Verified on a Windows + RTX 5070 Ti box: bare 'coli chat --model <g64>' now
prints '[GPU] auto-enabled CUDA ... 13.0 GB expert tier' and emits
COLI_CUDA=1 / COLI_GPUS=0 / CUDA_EXPERT_GB=13.044 (was: all unset, CPU-only).
Tests: 4 new cases (auto-enable, nvidia-smi-missing fallback, CPU-build
silent, Linux-unchanged) plus the 4 existing default-I/O tests guarded to
mock cuda_binary() so they stay host-independent. Full python suite green
(env_defaults 8, resource_plan 10, doctor 8, makefile_platform 3, cli_output 3).
Out of scope: doctor.cuda_linkage is also POSIX-only and mis-reports on
Windows — separate follow-up.
JustVugg caught a regression by running the PR: the code asked
`lscpu -p=CPU,Core,Socket` and read fields[1]/fields[2], but the comment's
claim was inverted -- `lscpu -p=<list>` emits EXACTLY the requested columns
(no CPU prefix), while bare `lscpu -p` prepends CPU.
On machines where the requested list is short or lscpu collapses to two
columns, every line was skipped by the `< 3` guard, cores stayed empty, and
the code fell through to os.cpu_count() -- the LOGICAL count. Result: 6
physical cores reported as 12 (SMT over-subscription), the opposite of the
fix.
Correct per review:
- ask lscpu for exactly 'core,socket'
- take fields[-2], fields[-1] (correct whether or not CPU is prepended)
- keep the warning scaffolding + the _resolve_physical_cores clamp
(those are the actual #325 fix; only the indexing was wrong)
Test now exercises BOTH the 2-column (-p=core,socket) and 3-column
(bare -p, CPU prefix) layouts, asserting 12 physical cores each. The
2-column case is the one that regressed: old parser returns 24, new
returns 12.
The core-count fix was necessary but not sufficient: @liangstein confirmed
physical_cpu_count() now returns 64 on his box, yet --auto-tier STILL pinned
decode to one core. A complete env diff between the plain (working) and
--auto-tier (broken) paths showed exactly three keys the plan adds:
OMP_NUM_THREADS = 64 (correct, verified)
OMP_PROC_BIND = spread
OMP_PLACES = cores
Since OMP_NUM_THREADS was already correct, the culprit is the affinity pair.
The mechanism: environment_for_plan() sets OMP_PROC_BIND=spread + OMP_PLACES=cores
in the launcher's env. The engine's hot-thread tuning (glm.c main, the
COLI_OMP_TUNED self-exec) then tries setenv("OMP_PROC_BIND","close", overwrite=0)
-- but overwrite=0 cannot replace an already-set var, so the plan's "spread"
wins. On the reporter's libgomp + multi-socket topology, spread + places=cores
collapsed the team to a single CPU even with 64 threads configured.
Fix: don't set affinity from the plan at all. The engine deliberately chose
"close" for cache locality (the tiny back-to-back per-expert matmuls want
adjacent cores), and the plain path already leaves affinity to the engine.
Removing the plan's spread/places makes --auto-tier match the working plain
path; a user wanting a specific policy can still set OMP_PROC_BIND/OMP_PLACES
in their own environment (environment_for_plan only setdefaults OMP_NUM_THREADS).
Verified via env diff: after the fix, --auto-tier adds ONLY OMP_NUM_THREADS
beyond the plain env -- the engine's own close/bind tuning now wins on both
paths identically.
Note: this could not be reproduced on Windows (MinGW libgomp prints "Affinity
not supported on this configuration" and ignores the vars entirely); it is
Linux-libgomp-specific, matching the reporter's Rocky 9 box.
Tests: rewrite test_applies_plan_without_overriding_explicit_settings to assert
the plan sets NO affinity vars on any platform (the old test encoded the buggy
platform-dependent spread/cores contract). Add
test_plan_does_not_set_omp_affinity_vars as a focused regression. 78/78 pass.
physical_cpu_count() silently returned 1 in two situations, and that value
flowed through build_plan -> OMP_NUM_THREADS to pin every matmul region to a
single thread under --auto-tier (reported on Rocky Linux 9, 512 GB RAM).
Two root causes:
1. The lscpu parse counted the wrong thing. `lscpu -p=core,socket` prepends the
CPU column, so the output is actually CPU,Core,Socket; the old set
comprehension collected (CPU,core,socket) tuples that were unique per logical
CPU. Now parse CPU,Core,Socket and dedupe on (core, socket) to get true
physical cores (the SMT-doubling the surrounding comments warn against was
the actual behavior).
2. Any probe failure fell through to `os.cpu_count() or 1`. On a cgroup'd or
otherwise constrained box os.cpu_count() can be 1 (or None), silently
capping the run. Skip offline core/socket fields ("-" instead of raising
ValueError) so a single offline row no longer discards the whole parse, and
replace the silent `or 1` fallback with os.cpu_count() (logical) plus an
explicit warning. Only return 1 when nothing at all is detected, and warn.
Also harden the win32 branch: declare argtypes/restype on
GetLogicalProcessorInformationEx (an undeclared 64-bit WinAPI returns c_int and
takes c_int pointers, so the probe could silently fail), and warn on its
fallbacks. Replace the silent max(1, int()) clamp in build_plan with
_resolve_physical_cores() that clamps to 1 with a warning instead of masking.
Tests: the existing OMP_NUM_THREADS test passed physical_cpus=24 explicitly, so
it never exercised the real probe -- that is why this regressed. Add regression
coverage for the lscpu physical-core dedup, offline fields, lscpu-missing
fallback, zero-cores degenerate case, and an end-to-end build_plan +
environment_for_plan check that OMP_NUM_THREADS reflects physical (not logical)
cores.
CI runs 'make check' = dependency-free tests, no model downloads (by design,
#140). glm_tiny/ is a gitignored generated fixture, so test_inefficiency.py
hard-failed on the Windows/macOS/Linux runners with 'config.json: No such file
or directory' instead of skipping.
_engine_present() now requires BOTH glm.exe AND glm_tiny/config.json, and
_skip_reason() names exactly which prerequisite is missing so the skip is
actionable. Verified: 8 skipped (0 failed) with the fixture absent; 5 pass +
3 CUDA-skip with it present.
Two layers of efficiency coverage for the engine, both parsing the telemetry
glm.c already emits (REPLAY/PROFILE/[PROF]/CUDA-tier) but nothing previously
asserted on:
1. test_inefficiency.py — tiny-model asserted regression tests (8 tests, run
in make test via test-python). Gate on: throughput floor, PROFILE phase
accounting sanity, disk-wait not dominant on a resident model, CPU greedy
determinism, and (when a CUDA build is present) CUDA init, dense VRAM
upload, and CPU-vs-CUDA argmax agreement >= 70%. CUDA tests auto-skip with
a clear build hint on CPU-only binaries.
2. test_efficiency_report.py — opt-in optimization dossier for a real model.
Turns on every instrumentation flag (PROF, COLI_CUDA_PROFILE, CACHE_ROUTE,
DISK_SPLIT, LOOKA) and prints 9 sections (provenance, throughput + tail
latency, where-time-goes, attention breakdown, expert cache, disk I/O +
phase split, routing quality + predictability, speculation, GPU tiers),
each flagging inefficiency with the concrete knob to move tok/s. Never
fails CI.
tools/efficiency.py is the shared harness: parse_run() captures every signal,
run_engine() wraps the subprocess. Reuses PROFILE_RE/SPEED_RE from
tools/benchmark_cuda_fixture.py and extends the tok/s regex to also catch the
run_text (parenthesized) format the full-model PROMPT path uses.
Makefile adds: efficiency / efficiency-cuda / efficiency-report targets.
Verified end-to-end on the full glm52_i4_g64 model (CPU + CUDA).
Upstream courtesy fix, found while auditing the conversion recipe for this
branch -- pre-existing in dev, not introduced by int3-g64, but directly
relevant to anyone converting for real (a #383-class gap: same failure
family as the resume/manifest work already done for --indir, and the
silent-mixing mechanism issue #355 fixed for a narrower case).
The --indir path already refuses to resume with different conversion
parameters on the same --outdir (a manifest records ebits/xbits/io_bits/
group_size/n_layers/bits_map and compares on every resume). The --repo
streaming download loops (main model, --mtp, --indexer) never got the
same guard: each shard's resume check is just `if os.path.exists(outp):
continue` -- true whether or not THIS run's flags match the flags that
produced that shard. A --repo conversion resumed with changed bits
(--xbits 3 -> 4 mid-run, say, after an interruption) would silently mix
bit-widths across shards in the same container, with no error and no log
line distinguishing it from a normal resume.
Fix: check_or_record_params(), a small shared helper mirroring the
--indir manifest's refuse-on-mismatch logic but without needing its
per-shard bookkeeping (the --repo loops already track shard completion
correctly via out-NNNNN.safetensors existence, since shard index maps
directly to output filename there -- only whether the params used SO FAR
still match needed adding). Applied to all three --repo loops with
per-mode sidecar files (.out-mtp-params.json / .out-idx-params.json /
.out-params.json) so a --mtp and a main-model conversion into the same
--outdir don't cross-check each other's parameters.
Also includes PROJ_BITS (the per-projection expert bit overrides) in the
tracked params dict on BOTH paths -- it was missing from --indir's
existing manifest too, so a resume with a changed --up-bits/--gate-bits/
--down-bits would have passed the existing guard silently.
Verified directly (no real HF downloads; --repo network paths can't be
exercised under this task's constraints): unit-tested
check_or_record_params() standalone -- fresh outdir accepts and records,
a same-params resume accepts, a changed --xbits is refused, and a
proj_bits-only change (nothing else different) is refused. Re-ran the
existing --indir dry-run end to end (convert, resume, resume-with-changed-
xbits) to confirm the manifest-based path still works correctly with
proj_bits added to its params dict.
Gates: make test-c (20/20) and make test-python (85/85) both pass.
Adds `-iq3` to the ablation harness: a faithful torch model of llama.cpp's
deployed 3.06-bpw IQ3_XXS format — 256-entry 4-dim magnitude grid
(extracted from ggml-common.h, MIT), signs factored per 8 weights with
the odd-parity constraint priced in (a violating block flips its
smallest-magnitude sign), fp16 super-scale per 256 + 4-bit sub-scale per
32 searched over all 16 codes. Nearest-grid search runs as chunked
matmul-argmin (|g|^2 - 2 q.g) — a full cdist materializes tens of GB on
a 100M-param tensor and OOMed the first run.
Measured (OLMoE-1B-7B, n=200 x hellaswag/arc/mmlu; the first four rows
reproduce the published ablations exactly):
fp16 58.0%
int4 per-row 48.7% (-9.3pp, the shipped container's scheme)
int3-g64 50.5% (-7.5pp)
int3-g64-e8-rot 51.5% (-6.5pp, simulated rate-scaled ball)
int3-iq3 49.3% (-8.7pp)
int3-iq3-rot 51.5% (-6.5pp)
The deployable IQ3 codebook plus rotation exactly ties the simulated E8
ball — that settles #452's codebook decision toward the IQ3-style block
structure, with rotation mandatory (worth 2.2pp on this codebook).
The VRAM-ranked prefix's host slabs are upload staging — read once and
freed right after — so binding them buys nothing and cost ~2 transient
VMAs each: measured PEAK maps 28,958 on the six-GPU host even with the
pin arenas in place. With the skip: 11,771, all of it the bounded LRU
ecache, which serves decode reads and stays correctly bound.
Root cause of #419's 'OOM slab': every per-slab mbind carries its own
memory policy, so bound regions cannot merge — measured ~2 VMAs per slab,
with or without MPOL_MF_MOVE. A PIN_GB=all load (19,456 experts x
slab+fslab) creates ~78k VMAs and crosses the default
vm.max_map_count=65530: posix_memalign dies with terabytes free.
The fix binds the pinned hot-store as ONE arena per layer. Experts of a
layer share a tensor shape, so a layer's pins pack at a fixed stride into
two arenas (weights + scales): 2 mbinds and a handful of VMAs per layer
instead of ~500. Slices are pre-attached to the slots before the load
loop — slab_cap covers expert_load's realloc check, so its alloc branch
never fires and expert_load itself is untouched. aslab marks arena
ownership: expert_host_release detaches instead of freeing (a REPIN
gpu-swap promotion must not free() an interior pointer), and
expert_host_ensure re-attaches the slice before reloading.
Per-slab mbind remains for the bounded allocations (dense qalloc, LRU
ecache, GPU-tier staging), now without MPOL_MF_MOVE: every bind lands
before the pread that first-touches the pages, so there is nothing to
migrate.
numa_init also gains a capability probe done right: one page-aligned
page (mbind rejects unaligned addresses with EINVAL), disabling only on
errno==EPERM — so a constrained container degrades with a message
instead of crashing later, and an EINVAL can never masquerade as a
missing capability.
The arena path activates only when interleave is actually on
(g_numa_nodes>=2, Linux, non-mmap): default builds stay byte-identical.
The grouped MoE kernels were per-row-only: GroupDesc had no group-size
fields, row_bytes() returned 0 for fmt=4, the scale buffer was hardcoded
to O floats, and a fmt=4 group that reached the generic path would have
been silently decoded as int2. This closed the GPU expert tier to every
grouped container — including the g64 quality line (#225) and the E8
lattice route (#347) whose whole point is fitting more experts in VRAM.
- ColiCudaTensor gains gs/scale_count; upload allocates O*ceil(I/gs)
scales for fmt=4 and applies the same offset->signed nibble conversion
as fmt=2 (identical packing). New ABI entry coli_cuda_tensor_upload_g
carries gs without touching the existing symbol — an old Windows DLL
missing it returns 0 and the tensor simply stays CPU-side.
- GroupDesc gains per-tensor group sizes; new grouped_hidden_g4_dual /
grouped_down_g4 apply the per-group scale inside the accumulation
(gs is required even, so a packed byte never straddles groups; gs=0
degrades to per-row, letting fmt=2 members ride the same launch).
- coli_cuda_expert_group routes any group containing fmt=4 through the
g4 kernels; pure-fmt=2 groups keep the existing paths byte-identical.
The generic fallback now explicitly rejects fmt=4 instead of decoding
garbage (#334's prevention note, made real).
tests/test_grouped_g4_cuda.cu: kernel-vs-CPU oracle over 50 trials x 3
experts — gs=64, a non-divisible tail group (200 % 64), and a per-row
member in the same launch: zero mismatches on a 5090.
make check 77/77; CPU, CUDA and MinGW builds clean.
rss_guard marked the victim slot eid=-1 under the lock, unlocked, and only then
freed s->slab. In that window the slot reads as {eid=-1, slab still valid} --
exactly the state pilot_realload's victim scan reuses first -- so a pilot worker
could claim it and pread into the slab while it was being freed: use-after-free,
or a double-free if the loader took its own realloc path.
Keep the free and the pointer/capacity NULLing inside the critical section, so
'slab valid' and 'slot reusable' are never simultaneously observable. The lock is
held across a free() (microseconds); workers only take it briefly for scan+reserve.
Reproduces on dev with a SINGLE pilot worker (rss_guard runs on the main thread
while the pilot worker runs in the background), whenever the RAM guard is active
(RSS_GUARD_GB, or any resolved g_ram_budget_gb).
make check 83/83, native + portable builds 0 warnings.
Re-derives the campaign's rtop8 breakthrough (fuse/rtop8-par @ b32439b,
commits 48b3a98 + b32439b) cleanly onto origin/dev @ 61004dc (post-#391
split), dropping the cb-chain commit (1130ac9) that branch was stacked
on. cb-chain touched only c/glm.c (now split into c/colibri.c); rtop8
touches only c/backend_metal.{h,mm} and c/tests/test_backend_metal.mm —
disjoint files, verified zero cb-chain remnants in this diff.
- r_top8_par: one SIMDGROUP per row (blocked lane ownership, taken
bitmask, shuffle-down argmax with ties->lower-index) replicating the
serial r_top8's first-max-wins ascending order exactly; topp/normk/
rscale tail verbatim on lane 0. Bench: serial 0.465 ms/layer (~55% of
the layer CB), parallel ~93x faster on the kernel, bit-exact.
- COLI_RTOP8 gate, default ON (renamed and inverted from the campaign's
COLI_RTOP8_PAR=0-default gate; COLI_RTOP8=0 is the opt-out escape).
- Expert-count generality (new hard requirement, REAP E=168 packages):
the parallel kernel's E<=256 contract is now enforced at EVERY call
site in host code (g_rtop8_width_ok / E<=256 checked before selecting
the pipeline), not just as an in-kernel defensive no-op -- closes a
latent gap where the campaign's SIMD-width guard only protected the
engine's automatic dispatch, not the standalone coli_metal_rtop8(par=1,
...) probe function metal-test itself uses. Out-of-contract requests
(E>256, or non-32-wide SIMD) now transparently run the serial kernel
instead of silently no-op'ing.
- metal-test: 16 new cases -- the campaign's 10-case exact-match fuzz
(now E-parametric) plus 6 new E-generality cases: E=168 (REAP) generic
+ massed-dup-ties, E=24 (<32 lane width) generic + ALL-EQUAL ties,
E=200 (a lane straddles the E boundary -- per=ceil(200/32)=7 doesn't
divide 200, so lane 28's ch[] block mixes 4 real indices with 3
sentinel ones; input is constructed to force those 4 into the top-8
deterministically, and the test asserts they were actually selected,
not just that serial==parallel), and E=257 (out-of-contract, proves
the auto-serial-fallback engages: this case is confirmed load-bearing
-- it fails without the new host-side guard). 15 stock + 10 ported +
6 new = 31 total metal-test cases.
Builds (METAL=0/1) and full C/python suites verified unchanged vs stock
61004dc (0 new warnings, identical pass counts). On-box A/B on this
branch is pending (see PR_BODY.md) -- no model runs performed here.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Under O_DIRECT the expert *weights* are read with the direct fd (bypassing the
page cache), so a POSIX_FADV_WILLNEED on them warms pages the demand read never
consumes -- pure wasted readahead on the disk, the scarcest resource when
streaming. The .qs scales are ALWAYS read buffered (pread on the normal fd), so
keep their WILLNEED. This only changes hint-only PILOT (PILOT=1, PILOT_REAL=0)
combined with DIRECT=1; fadvise is advisory, so output is bit-identical.
Prepared, not yet measured: needs a real NVMe + full model to A/B cold-stream
tok/s (PILOT=1 DIRECT=1, with/without this patch, TEMP=0). See #441.
pin_load computed npin from the RAM budget FIRST, then carved the
VRAM-ranked prefix out of those npin slots. With CUDA_RELEASE_HOST the
prefix's host slabs are freed right after upload — so on a multi-GPU
host the top-ranked experts consumed the RAM budget without occupying
RAM, and the CPU tier pinned only the leftovers. Measured on 6x RTX 5090
(251 GB): 9,280 VRAM + only 1,721 RAM pins (32.5 GB warm) on a box whose
RAM fits ~10k more — the cold tail then paid disk on every token and the
hit rate ceilinged at 99.0% forever.
Move the VRAM budget estimate above the npin finalization and make the
release-destined prefix ADDITIVE to the RAM-derived count. Same box,
same env plus the fix:
[PIN] placement: 9,280 VRAM + 10,176 RAM (192.5 GB warm)
expert hit rate 100.0% (pin 100.0% + lru 0.0%) — disk 0
1,024-token greedy decode: 3.62 -> 6.21 tok/s (+72%)
warm late segment (t=768-1024): 5.11 -> 5.73 tok/s (+12%)
prefill: 10.4 -> 8.8 s
The whole-run gain is the LRU warmup penalty disappearing (full
residency from the first token); the late-segment gain is the steady
~0.9%-miss disk residue recovered. Non-release configs are untouched:
prefix_est stays 0 and the arithmetic reduces to today's exactly.
The __AVXVNNI__ branches of dot_i8i8/dot_i4i8 (quant.h) accumulated every vpdpbusd
into a single register, a serial dependency chain (vpdpbusd is latency-bound ~5c).
Use 4 independent accumulators, mirroring the NEON 4-accumulator path in this file.
On x86 the tiled SMMLA _mm path is ARM-only, so these two dots are THE int8/int4
IDOT matmul kernels on x86 (g_idot=1 default), prefill and decode.
Integer accumulation is associative -> bit-identical (test_idot passes bit-for-bit).
Microbench (tests/bench_idot, i7-12700H P-core, I=6144):
dot_i8i8 6.85 -> 19.26 GB/s (2.81x)
dot_i4i8 3.41 -> 7.25 GB/s (2.13x)
FABIOTESS