Upstream courtesy fix, found while auditing the conversion recipe for this
branch -- pre-existing in dev, not introduced by int3-g64, but directly
relevant to anyone converting for real (a #383-class gap: same failure
family as the resume/manifest work already done for --indir, and the
silent-mixing mechanism issue #355 fixed for a narrower case).
The --indir path already refuses to resume with different conversion
parameters on the same --outdir (a manifest records ebits/xbits/io_bits/
group_size/n_layers/bits_map and compares on every resume). The --repo
streaming download loops (main model, --mtp, --indexer) never got the
same guard: each shard's resume check is just `if os.path.exists(outp):
continue` -- true whether or not THIS run's flags match the flags that
produced that shard. A --repo conversion resumed with changed bits
(--xbits 3 -> 4 mid-run, say, after an interruption) would silently mix
bit-widths across shards in the same container, with no error and no log
line distinguishing it from a normal resume.
Fix: check_or_record_params(), a small shared helper mirroring the
--indir manifest's refuse-on-mismatch logic but without needing its
per-shard bookkeeping (the --repo loops already track shard completion
correctly via out-NNNNN.safetensors existence, since shard index maps
directly to output filename there -- only whether the params used SO FAR
still match needed adding). Applied to all three --repo loops with
per-mode sidecar files (.out-mtp-params.json / .out-idx-params.json /
.out-params.json) so a --mtp and a main-model conversion into the same
--outdir don't cross-check each other's parameters.
Also includes PROJ_BITS (the per-projection expert bit overrides) in the
tracked params dict on BOTH paths -- it was missing from --indir's
existing manifest too, so a resume with a changed --up-bits/--gate-bits/
--down-bits would have passed the existing guard silently.
Verified directly (no real HF downloads; --repo network paths can't be
exercised under this task's constraints): unit-tested
check_or_record_params() standalone -- fresh outdir accepts and records,
a same-params resume accepts, a changed --xbits is refused, and a
proj_bits-only change (nothing else different) is refused. Re-ran the
existing --indir dry-run end to end (convert, resume, resume-with-changed-
xbits) to confirm the manifest-based path still works correctly with
proj_bits added to its params dict.
Gates: make test-c (20/20) and make test-python (85/85) both pass.
Adds `-iq3` to the ablation harness: a faithful torch model of llama.cpp's
deployed 3.06-bpw IQ3_XXS format — 256-entry 4-dim magnitude grid
(extracted from ggml-common.h, MIT), signs factored per 8 weights with
the odd-parity constraint priced in (a violating block flips its
smallest-magnitude sign), fp16 super-scale per 256 + 4-bit sub-scale per
32 searched over all 16 codes. Nearest-grid search runs as chunked
matmul-argmin (|g|^2 - 2 q.g) — a full cdist materializes tens of GB on
a 100M-param tensor and OOMed the first run.
Measured (OLMoE-1B-7B, n=200 x hellaswag/arc/mmlu; the first four rows
reproduce the published ablations exactly):
fp16 58.0%
int4 per-row 48.7% (-9.3pp, the shipped container's scheme)
int3-g64 50.5% (-7.5pp)
int3-g64-e8-rot 51.5% (-6.5pp, simulated rate-scaled ball)
int3-iq3 49.3% (-8.7pp)
int3-iq3-rot 51.5% (-6.5pp)
The deployable IQ3 codebook plus rotation exactly ties the simulated E8
ball — that settles #452's codebook decision toward the IQ3-style block
structure, with rotation mandatory (worth 2.2pp on this codebook).
The VRAM-ranked prefix's host slabs are upload staging — read once and
freed right after — so binding them buys nothing and cost ~2 transient
VMAs each: measured PEAK maps 28,958 on the six-GPU host even with the
pin arenas in place. With the skip: 11,771, all of it the bounded LRU
ecache, which serves decode reads and stays correctly bound.
Root cause of #419's 'OOM slab': every per-slab mbind carries its own
memory policy, so bound regions cannot merge — measured ~2 VMAs per slab,
with or without MPOL_MF_MOVE. A PIN_GB=all load (19,456 experts x
slab+fslab) creates ~78k VMAs and crosses the default
vm.max_map_count=65530: posix_memalign dies with terabytes free.
The fix binds the pinned hot-store as ONE arena per layer. Experts of a
layer share a tensor shape, so a layer's pins pack at a fixed stride into
two arenas (weights + scales): 2 mbinds and a handful of VMAs per layer
instead of ~500. Slices are pre-attached to the slots before the load
loop — slab_cap covers expert_load's realloc check, so its alloc branch
never fires and expert_load itself is untouched. aslab marks arena
ownership: expert_host_release detaches instead of freeing (a REPIN
gpu-swap promotion must not free() an interior pointer), and
expert_host_ensure re-attaches the slice before reloading.
Per-slab mbind remains for the bounded allocations (dense qalloc, LRU
ecache, GPU-tier staging), now without MPOL_MF_MOVE: every bind lands
before the pread that first-touches the pages, so there is nothing to
migrate.
numa_init also gains a capability probe done right: one page-aligned
page (mbind rejects unaligned addresses with EINVAL), disabling only on
errno==EPERM — so a constrained container degrades with a message
instead of crashing later, and an EINVAL can never masquerade as a
missing capability.
The arena path activates only when interleave is actually on
(g_numa_nodes>=2, Linux, non-mmap): default builds stay byte-identical.
The grouped MoE kernels were per-row-only: GroupDesc had no group-size
fields, row_bytes() returned 0 for fmt=4, the scale buffer was hardcoded
to O floats, and a fmt=4 group that reached the generic path would have
been silently decoded as int2. This closed the GPU expert tier to every
grouped container — including the g64 quality line (#225) and the E8
lattice route (#347) whose whole point is fitting more experts in VRAM.
- ColiCudaTensor gains gs/scale_count; upload allocates O*ceil(I/gs)
scales for fmt=4 and applies the same offset->signed nibble conversion
as fmt=2 (identical packing). New ABI entry coli_cuda_tensor_upload_g
carries gs without touching the existing symbol — an old Windows DLL
missing it returns 0 and the tensor simply stays CPU-side.
- GroupDesc gains per-tensor group sizes; new grouped_hidden_g4_dual /
grouped_down_g4 apply the per-group scale inside the accumulation
(gs is required even, so a packed byte never straddles groups; gs=0
degrades to per-row, letting fmt=2 members ride the same launch).
- coli_cuda_expert_group routes any group containing fmt=4 through the
g4 kernels; pure-fmt=2 groups keep the existing paths byte-identical.
The generic fallback now explicitly rejects fmt=4 instead of decoding
garbage (#334's prevention note, made real).
tests/test_grouped_g4_cuda.cu: kernel-vs-CPU oracle over 50 trials x 3
experts — gs=64, a non-divisible tail group (200 % 64), and a per-row
member in the same launch: zero mismatches on a 5090.
make check 77/77; CPU, CUDA and MinGW builds clean.
rss_guard marked the victim slot eid=-1 under the lock, unlocked, and only then
freed s->slab. In that window the slot reads as {eid=-1, slab still valid} --
exactly the state pilot_realload's victim scan reuses first -- so a pilot worker
could claim it and pread into the slab while it was being freed: use-after-free,
or a double-free if the loader took its own realloc path.
Keep the free and the pointer/capacity NULLing inside the critical section, so
'slab valid' and 'slot reusable' are never simultaneously observable. The lock is
held across a free() (microseconds); workers only take it briefly for scan+reserve.
Reproduces on dev with a SINGLE pilot worker (rss_guard runs on the main thread
while the pilot worker runs in the background), whenever the RAM guard is active
(RSS_GUARD_GB, or any resolved g_ram_budget_gb).
make check 83/83, native + portable builds 0 warnings.
Re-derives the campaign's rtop8 breakthrough (fuse/rtop8-par @ b32439b,
commits 48b3a98 + b32439b) cleanly onto origin/dev @ 61004dc (post-#391
split), dropping the cb-chain commit (1130ac9) that branch was stacked
on. cb-chain touched only c/glm.c (now split into c/colibri.c); rtop8
touches only c/backend_metal.{h,mm} and c/tests/test_backend_metal.mm —
disjoint files, verified zero cb-chain remnants in this diff.
- r_top8_par: one SIMDGROUP per row (blocked lane ownership, taken
bitmask, shuffle-down argmax with ties->lower-index) replicating the
serial r_top8's first-max-wins ascending order exactly; topp/normk/
rscale tail verbatim on lane 0. Bench: serial 0.465 ms/layer (~55% of
the layer CB), parallel ~93x faster on the kernel, bit-exact.
- COLI_RTOP8 gate, default ON (renamed and inverted from the campaign's
COLI_RTOP8_PAR=0-default gate; COLI_RTOP8=0 is the opt-out escape).
- Expert-count generality (new hard requirement, REAP E=168 packages):
the parallel kernel's E<=256 contract is now enforced at EVERY call
site in host code (g_rtop8_width_ok / E<=256 checked before selecting
the pipeline), not just as an in-kernel defensive no-op -- closes a
latent gap where the campaign's SIMD-width guard only protected the
engine's automatic dispatch, not the standalone coli_metal_rtop8(par=1,
...) probe function metal-test itself uses. Out-of-contract requests
(E>256, or non-32-wide SIMD) now transparently run the serial kernel
instead of silently no-op'ing.
- metal-test: 16 new cases -- the campaign's 10-case exact-match fuzz
(now E-parametric) plus 6 new E-generality cases: E=168 (REAP) generic
+ massed-dup-ties, E=24 (<32 lane width) generic + ALL-EQUAL ties,
E=200 (a lane straddles the E boundary -- per=ceil(200/32)=7 doesn't
divide 200, so lane 28's ch[] block mixes 4 real indices with 3
sentinel ones; input is constructed to force those 4 into the top-8
deterministically, and the test asserts they were actually selected,
not just that serial==parallel), and E=257 (out-of-contract, proves
the auto-serial-fallback engages: this case is confirmed load-bearing
-- it fails without the new host-side guard). 15 stock + 10 ported +
6 new = 31 total metal-test cases.
Builds (METAL=0/1) and full C/python suites verified unchanged vs stock
61004dc (0 new warnings, identical pass counts). On-box A/B on this
branch is pending (see PR_BODY.md) -- no model runs performed here.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Under O_DIRECT the expert *weights* are read with the direct fd (bypassing the
page cache), so a POSIX_FADV_WILLNEED on them warms pages the demand read never
consumes -- pure wasted readahead on the disk, the scarcest resource when
streaming. The .qs scales are ALWAYS read buffered (pread on the normal fd), so
keep their WILLNEED. This only changes hint-only PILOT (PILOT=1, PILOT_REAL=0)
combined with DIRECT=1; fadvise is advisory, so output is bit-identical.
Prepared, not yet measured: needs a real NVMe + full model to A/B cold-stream
tok/s (PILOT=1 DIRECT=1, with/without this patch, TEMP=0). See #441.
pin_load computed npin from the RAM budget FIRST, then carved the
VRAM-ranked prefix out of those npin slots. With CUDA_RELEASE_HOST the
prefix's host slabs are freed right after upload — so on a multi-GPU
host the top-ranked experts consumed the RAM budget without occupying
RAM, and the CPU tier pinned only the leftovers. Measured on 6x RTX 5090
(251 GB): 9,280 VRAM + only 1,721 RAM pins (32.5 GB warm) on a box whose
RAM fits ~10k more — the cold tail then paid disk on every token and the
hit rate ceilinged at 99.0% forever.
Move the VRAM budget estimate above the npin finalization and make the
release-destined prefix ADDITIVE to the RAM-derived count. Same box,
same env plus the fix:
[PIN] placement: 9,280 VRAM + 10,176 RAM (192.5 GB warm)
expert hit rate 100.0% (pin 100.0% + lru 0.0%) — disk 0
1,024-token greedy decode: 3.62 -> 6.21 tok/s (+72%)
warm late segment (t=768-1024): 5.11 -> 5.73 tok/s (+12%)
prefill: 10.4 -> 8.8 s
The whole-run gain is the LRU warmup penalty disappearing (full
residency from the first token); the late-segment gain is the steady
~0.9%-miss disk residue recovered. Non-release configs are untouched:
prefix_est stays 0 and the arithmetic reduces to today's exactly.
The __AVXVNNI__ branches of dot_i8i8/dot_i4i8 (quant.h) accumulated every vpdpbusd
into a single register, a serial dependency chain (vpdpbusd is latency-bound ~5c).
Use 4 independent accumulators, mirroring the NEON 4-accumulator path in this file.
On x86 the tiled SMMLA _mm path is ARM-only, so these two dots are THE int8/int4
IDOT matmul kernels on x86 (g_idot=1 default), prefill and decode.
Integer accumulation is associative -> bit-identical (test_idot passes bit-for-bit).
Microbench (tests/bench_idot, i7-12700H P-core, I=6144):
dot_i8i8 6.85 -> 19.26 GB/s (2.81x)
dot_i4i8 3.41 -> 7.25 GB/s (2.13x)
Re-derivation of e7/disk-class-instr @ de6dd6d (base caa49f7) onto
origin/dev @ 61004dc, after PR #391 split c/glm.c into c/colibri.c plus
quant.h/sample.h/kv_persist.h/telemetry.h/grammar.h. Semantic equivalence,
not a copy: same events, same accounting, re-sited onto the new tree.
Placement: colibri.c, unchanged from before the split. telemetry.h (#391)
is the dashboard/stats module (HWINFO/TIERS/EMAP/HITS protocol lines +
usage persistence) — prof_report(), expert_load_impl(), moe(), g_prof_io
and g_edisk_ns all stayed in colibri.c, so DISK-CLASS's aggregation and
printing follow them there. No relocation needed, no DEVIATION.
Read path: #362 (prefetcher-v3, 22509fc) turned out to be testbed-only
scope per its own merge message ("glm.c untouched") — confirmed zero
c/glm.c changes in that merge's diff. The seven expert_load() call sites
this patch touches (pipe_worker, expert_host_ensure, moe()'s OMP miss
loop, pilot_realload, repin_pass_limit, pin_load x2) are structurally
identical to the pre-split tree; the demand flag re-attaches at the same
sites with the same semantics (1 only at moe()'s own PIPE/OMP miss path,
0 everywhere else), so DISK-CLASS still counts demand loads only.
One real drift, unrelated to #362: #417 (cfcc742) fixed the exact "Metal
pre-routed FASE A never bumps the real elast/eaccess_clock" defect this
feature's comments described as a documented, deliberately-unfixed
upstream issue — the real clock now ticks in FASE A too. The private
elast_dc/eaccess_clock_dc clock is kept anyway: its job was never only
to route around that freeze, it also snapshots pre-bump state so a
call's own routing bump can't contaminate its own classification, and
keeping DISK-CLASS's bookkeeping fully separate from stock elast state
is what makes "byte-identical with PROF=0" provable by construction
instead of by argument. Code comments referencing the old defect are
updated to reflect the fix (historical note + #417/cfcc742 pointer)
rather than describing a bug that no longer exists.
Gates: make glm METAL=0 and METAL=1 both clean, zero warnings (matches
stock 61004dc, also built clean with zero warnings for comparison).
make test-c: 0 failures. make test-python: 77 tests, OK.
Authored by Fable 5 in Claude Code, analysis in partnership with
@Monotophic.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
GLM-5.2's config defines three stop tokens: <|endoftext|>, <|user|>, and
<|observation|>. In serve mode, when the model generates <tool_call> blocks,
int4-quantized logit noise can cause argmax to pick a <|user|> or
<|observation|> token ID, immediately stopping generation.
The <|user|> and <|observation|> tokens are role markers handled by the
Python API server, not the C engine. Filter them out in stops_arm() when
SERVE=1, keeping only <|endoftext|> as the stop token.
- Add SERVE-mode guard in stops_arm() that retains only the EOS token
- Log the number of filtered tokens for diagnostics
The generation() method was re-extracting tools from the raw request body,
bypassing the filtering that render_chat() applies when tool_choice is a
function object (forced function call). This caused parse_tool_calls() to
receive the unfiltered tool list, producing incorrect or missing tool_calls
in the response.
- Add tools and tool_choice parameters to generation() signature
- Pass them from chat_completion() after render_chat() processing
- Add early structural validation of tools in generation_options()
for clear HTTP 400 errors on malformed input
The TC_W4A16 branch of coli_cuda_expert_group handled every expert in a
per-expert loop: rows >= threshold got the Tensor Core path, everything
below fell back to 4 naive launches per expert. At decode every expert
has 1 row, so the whole group rode the fallback — ~981 quant_matmul
micro-launches per token (#431's measured flood) — while the grouped
3-launch path (grouped_hidden_w4_dual + silu + grouped_down_w4) sat one
else-if below, unreachable whenever the PREFILL tuning flag was set.
Gate the branch on 'at least one expert reaches the TC row threshold':
all-small groups (decode) now fall through to the grouped kernels.
Measured on 6x RTX 5090 (full residency, 39 forwards under nsys):
expert-side quant_matmul instances drop 981 -> 337 per forward, total
launches ~1,490 -> ~850 per token (-43%). Wall-clock is parity at the
A/B operating point — the win is structural (PR-C graph node count,
launch-tax share at champion speed).
Behavioural fix folded in: before this change, toggling TC_W4A16 — a
prefill-only optimization — changed DECODE output text (kernel-family
divergence, #100 class). After it, decode always uses the grouped
family: TC_W4A16=1 and =0 now produce byte-identical decode text
(verified, 96-token greedy A/B), and the flag affects only the prefill
it was built for.
First increment of the #431 plan (device router -> indirect kernels ->
one-graph decode). At decode (S=1) on the pipe2 path, the router runs on
the layer's home device: a tiny E x D logits GEMV + sigmoid, then a
single-thread selection kernel that clones moe()'s plain routing path
verbatim — bias-augmented top-K by choice with strict-> tie-breaking,
weights from the raw logit, route-level TOPP truncation, norm_topk,
routed_scale. Results pack into one scratch buffer and come back in a
single ~68-byte D2H; moe() consumes them through the same pre-routed
shortcut the Metal layer-CB uses (g_pre_idx, #417 bookkeeping included),
so usage/heat/recency accounting is identical to the CPU router.
Structural value: routing becomes available ON the device timeline,
which is what PR-B (indirect expert kernels, static topology) and PR-C
(whole-decode CUDA Graph) build on.
Opt-in, default off. Gated to the plain routing path — CACHE_ROUTE,
ROUTE_P and ROUTE_TRACE keep the CPU ranking they need; any upload or
launch failure falls back to the CPU router silently. Router weights
(E x D f32, ~6.3 MB/layer) upload lazily to the layer's home device.
tests/test_router_cuda.cu: kernel-vs-CPU-reference oracle over 200
random trials (mixed TOPP/norm_topk/scale): 200/200 exact selections,
zero near-tie flips, zero weight mismatches on a 5090.
Update the rebased documentation and comments to name colibri.c after the #391 coordinator split, and correct the E5 comparison table to reflect its instrumentation and OOM-unwind touches.
Co-authored-by: Christopher Brand <brand.christopher.c@gmail.com>
1. DEFENSIVE (backend_metal.mm, coli_metal_register): re-registering a live
base overwrote s.buf and resset_add'ed the new wrapper without removing
the OLD one from the residency set -- ARC drops our reference but the set
retains the object and keeps its pages resident: a leak and a
set/g_slabs divergence. The found branch now stashes the replaced
wrapper under g_slab_mtx and, after dropping the lock (round-1 invariant:
no Metal call under the slab lock), resset_remove(old)s it before
resset_add(b); identical old==b early-outs both set operations.
Invariant defended, stated in the comment: residency-set membership
mirrors g_slabs exactly. No in-tree caller re-registers a live base
today (audited) -- closed defensively.
2. DOC (SUMMARY.md): the moe_submit lifecycle bullet still said
resset_flush commits "under g_slab_mtx" -- stale text from before the
round-1 mutex split; the code takes g_resset_mtx and never touches the
slab lock there. Parenthetical corrected; register bullet updated to
describe the re-register hygiene from item 1.
expert_load's fslab-OOM unwind freed s->slab via compat_aligned_free without
coli_metal_unregister -- a pre-existing gap on main/dev (validator-confirmed
during E4) that leaves a stale g_slabs entry over freed memory, letting
resolve() hand the GPU a dangling pointer. Under COLI_METAL_RESSET=1 the
exposure is strictly longer-lived: the wrapped buffer would stay a permanent
residency-set member over the freed pages instead of stock's transient
last-command-buffer window. Ported from e4/metal-heap validator fix 6753225,
adapted to dev's non-heap code shape (no coli_metal_heap_free wrapper --
plain unregister-before-free).
The uring_load_add analog (E4 audit round-2 insurance) is deliberately not
carried: that arm is #ifdef __linux__-gated while COLI_METAL is macOS-only,
so it is dead code on every real build target, and unlike E4 this branch has
no allocation-path reason to touch the function.
Reachable only through allocation failure mid-load; verified by inspection
and clean builds (no OOM-injection harness in tree). SUMMARY.md item 10
updated to "carried on this branch".
1. REQUIRED (backend_metal.mm): no Metal call runs under g_slab_mtx anymore.
The set mutations (addAllocation/removeAllocation/commit) and the dirty
flag moved to a dedicated g_resset_mtx; coli_metal_register/unregister do
their g_slabs bookkeeping under g_slab_mtx exactly as stock, then call
resset_add/resset_remove after dropping it (still before returning, so
unregister's remove+commit still lands before the caller frees the host
memory). The original shape -- commit under the slab lock the parallel OMP
loader threads contend on -- was structurally identical to the mutex-over-
live-Metal-call bug E4's audit round 2 fixed, the leading suspect for its
replicated +12s expert-disk regression. The register->flush->resolve
happens-before argument survives the split: resset_add completes inside
coli_metal_register before it returns, the engine cannot dispatch an
expert before its load returns, and add/flush are serialized by
g_resset_mtx (comment at resset_add). The two mutexes are never held
together, so no lock-order hazard exists.
2. REQUIRED (backend_metal.mm, backend_metal.h, glm.c): the resset_flush
cost in moe_submit -- deliberately outside the g_t_setup window so the
A/B harness counters keep their meaning -- was invisible. New
g_t_resset_flush accumulator timed around the flush, exported via
coli_metal_resset_stats(), printed by profile_print as a separate
gate-on-only "METAL-RESSET: flush" line (mirrors E4's METAL-HEAP line;
the METAL: line the harness parses keeps its exact format; stock output
byte-identical -- the stats call returns 0 with the gate off and the
whole block sits in the pre-existing #ifdef COLI_METAL arm). Register-
side add/remove costs land in the existing t_ewait window; comment at
resset_add says so instead of a second counter.
3. REQUIRED (SUMMARY.md; the moe_submit commit message was already reworded
in place, pre-push): corrected two false attributions -- the hazard-
tracking and thread-safety statements were credited to the SDK header,
which is silent on both; the actual source is Apple's online
MTLResidencySet class reference and residency-set adoption guide
(fetched 2026-07-18). Note: the scaffolding commit's message was checked
and contains no such claim, so it was left untouched.
4. DOC (SUMMARY.md): the pre-existing fslab OOM-unwind bug (glm.c ~1868-73,
frees s->slab without unregister) has a strictly longer-lived exposure
under E5 (permanent set member vs stock's transient per-CB declaration).
Out of scope here; the upstream PR built from E5 must carry the one-line
fix (reference: E4 branch commit 6753225).
The one seam the mechanism history actually implicates: moe_submit's `use`
vector (resolved expert weight/scale slabs) is the only useResource: loop
whose length scales with LRU cache size. With COLI_METAL_RESSET=1, skip that
loop entirely -- the queue-attached MTLResidencySet already guarantees those
buffers are resident -- after resset_flush() commits any pending adds from a
loader burst. Every other useResource: call site (bind_gemv's weight/scale
buffers, attn_decode/layer_decode's Lb/Rb/kvbW/kvbS/inB/pnB/rwB/rbB,
coli_metal_gemm's wb/sb) is left unconditionally unchanged: none of them
scale with cache size, and Lb/Rb carry real GPU-side write traffic ordered by
existing explicit memoryBarrierWithScope: calls, not by useResource:'s hazard
tracking -- narrowing the blast radius rather than removing useResource:
uniformly. Full reasoning, the hazard-tracking tradeoff (residency sets don't
support hazard tracking, per Apple's MTLResidencySet developer documentation
and residency-set adoption guide; the SDK header itself is silent on the
topic), and every judgment call in UNCERTAINTIES: see SUMMARY.md.
Gate off is unaffected: g_resset_enabled is false, so the useResource: loop
runs exactly as before.
Env-gated (COLI_METAL_RESSET=1) persistent MTLResidencySet attached to g_queue
(macOS 15+, @available-guarded with a one-line stderr fallback). Adds
resset_add/resset_remove/resset_flush helpers, wires them into the existing
coli_metal_register/coli_metal_unregister/coli_metal_shutdown bodies -- no new
functions in backend_metal.h, no glm.c changes, every existing call site keeps
its current signature and behavior.
register() defers the commit (resset_add just marks dirty, under the same
g_slab_mtx that already serializes parallel OMP loader threads) so a loader
burst doesn't pay a commit per slab; unregister() commits synchronously and
immediately, because the caller frees the underlying host memory right after
it returns and a deferred removal would leave the set referencing freed
memory. Nothing yet reads g_resset_enabled to change dispatch behavior --
this commit is bookkeeping only, gate on or off, so it does not change what
any command buffer does (verified by inspection: no useResource:/useHeap:
call site is touched here).
Gate off (default) is byte-for-byte the stock path: g_resset_enabled starts
false and nothing sets it outside the COLI_METAL_RESSET branch in
coli_metal_init, so every new helper is a no-op.
See SUMMARY.md (next commit) for the full design and UNCERTAINTIES.
The has_mtp completeness probe checked for `mlp.experts.255.down_proj.weight`,
which only exists when n_routed_experts == 256. REAP-pruned checkpoints (and any
MoE with a different expert count) have fewer experts, so the probe spuriously
reported has_mtp=0 and disabled MTP speculative decode even though the head was
present and complete. Probe `mlp.experts.<n_experts-1>` instead.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A second (read-only) copy of the model on another drive is registered as a
per-shard read replica; expert loads are split between the drives by a
deterministic (layer,eid) hash weighted by COLI_DISK_WEIGHTS=<primary>,<mirror>
or, by default, by a startup bandwidth probe using the engine's own access
pattern (parallel ~19 MB reads, O_DIRECT). Cold decode is disk-bound
(~11 GB/token), so two NVMe queues add up.
- st.h: mirror accepted per file only if size + safetensors header are
byte-identical to the primary (identical data_offsets by construction, so
every pread is valid on either copy); partial mirrors work (smaller second
SSD holding only some shards); the mirror is never written — .coli_usage /
.coli_kv stay on the primary.
- glm.c: routing covers the coalesced slab pread, O_DIRECT, mmap/Metal and
scale reads, plus the OMP-parallel pin/autopin warmup (streams from both
drives). Deterministic routing keeps readahead/PILOT WILLNEED on the same
drive as the demand read and avoids caching an expert twice. A mirror read
error falls back to the primary (one warning, no crash). Per-drive bytes
are reported in a MIRROR: stats line.
- tests: test_st_mirror covers validation, read equality on both replicas,
and the rejection paths (divergent header, size mismatch, missing file).
Measured on 2x NVMe (GLM-5.2 int4, greedy, DRAFT=0, DIRECT=1, cold-ish
cache): 0.42 -> 0.57 tok/s (+36%), expert-disk service 15.2s -> 10.3s,
byte-identical output; probe chose a 48/52 split.
Extend resource_plan to classify the hardware into bottleneck regimes
(disk / memory / mixed / compute) and derive tuning knobs automatically:
MTP: off when compute-bound (42% loss at full residency, #389)
or disk-bound with <90% hit (union growth adds reads)
PIPE: COLI_CUDA_PIPE=1 single-GPU, =2 multi-GPU, PIPE=1 CPU disk
NUMA: selective interleave for GPU hosts, blanket hint for CPU-only
PIN: PIN_GB=all when fully resident + no GPU
OMP: COLI_NO_OMP_TUNE=1 for Metal (spin steals GPU power)
`coli plan` now shows an auto-tune section with each knob and its
reason. `environment_for_plan()` applies them via setdefault so
explicit user settings always win.
plan version stays at 2 (additive fields: bottleneck_class,
projected_hit_rate, tune). 7 new tests covering all regimes.
Rename glm.c → colibri.c and extract four self-contained modules
into header-only files (same pattern as st.h/tier.h/grammar.h):
quant.h (672 lines) — SIMD matmul kernels, quantization
sample.h (143 lines) — RNG, top-p sampling, stop-set
kv_persist.h (121 lines) — .coli_kv disk persistence
telemetry.h (189 lines) — dashboard protocol, stats, usage
Main engine file shrinks from 6588 to 5396 lines (−18%).
Build system: primary target is now colibri$(EXE); `make glm`
remains as a phony alias for backward compat. CI, setup.sh,
coli CLI, and all 10 test files that include the engine are
updated. make check passes (C + Python, 73 tests, zero warnings).
On Metal, when routing is precomputed on the GPU (g_pre_idx), the moe fast
path bumps eusage/ehit/eheat for the selected experts but skips the one thing
the full CPU router does at the equivalent site: elast[layer][e] =
++eaccess_clock. So the session-local recency clock advances during prefill
(full router) but freezes the moment GPU-prerouted decode starts, and REPIN's
tier_pick_lfru() tie-breaker then runs on stale recency for the rest of the
run. Mirror the exact update the non-Metal path already does. Inside
#ifdef COLI_METAL, so CPU/CUDA are untouched; elast only feeds the LFRU
eviction heuristic, so this cannot affect output, only which experts REPIN
keeps warm.
Found and reported by @monotophic with a source-level trace repro
(ELAST_TRACE). Fix is inspection-verified against line ~3055; needs a
Metal build to exercise end-to-end.
Closes#417
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Colibri loads model directories and safetensors from mirrors it does not
control, so the file's declared shapes and byte spans are attacker-influenced
input. Three memory-safety holes on that boundary, independently confirmed
(incl. a from-scratch adversarial audit that re-derived the same two) and
present in the shipped v1.0.0:
- st.h st_read_f32: numel came from the shape, nbytes from the offsets, with
no cross-check. A crafted tensor whose shape inflates numel past nbytes made
the BF16/F16 loop read past the malloc'd raw buffer and the F32 memcpy write
past the caller's config-sized destination (heap OOB read + write). Now
enforce numel*esz == nbytes before any copy.
- st.h header parse: the shape product could overflow int64 to a small/negative
numel that would then pass the cross-check. Guard each multiply.
- glm.c qt_resolve_fmt (new, replaces the three duplicated "?1:?2:3" fmt sites
in qt_from_disk and both expert_load arms): the old inference SILENTLY fell
to int2 for any unrecognized weight byte count, so a too-short weight became
a valid int2 whose matmul read O*I nibbles past the buffer; and an oversized
scale array overflowed the per-row t->s. Now the weight bytes must match a
known int8/int4/int2 layout and the scale array must match the expected
per-row (O) or grouped (O*ng) cardinality, else refuse.
- glm.c config/generation_config slurp: unbounded ftell -> malloc(n+1) gave a
hostile file a load-time OOM, and on malloc failure b[got]=0 was a NULL
deref. Cap at 256 MB and NULL-check.
Verified: TF token-exactness unchanged on every quant format (full-precision
32/32, int4 11/32, int2 1/32, mix 5/32 -- byte-identical to the pre-change
binary); fmt=4 grouped path preserved (the scale check is by construction the
same condition detect_group_size already imposed); a hand-crafted hostile
safetensors is refused cleanly; ASan+UBSan clean on legit and hostile loads
(only the pre-existing intentional startup leaks remain).
These are the C trust-boundary items of #368, landed as a minimal standalone
fix; the server-side and build items of that PR follow via its rebase.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
cap_for_ram's projection is an estimate: on the GB10 (#403) long generations
overshot it by ~40 GB (projected 74.4, real 115.6) and the kernel killed the
engine three times. Run D of the issue proves a low cap CONTAINS the growth;
this guard does that automatically, keyed on MEASURED RSS instead of the
projection.
At the repin safe point (no moe in flight), every ~16 emitted tokens: if RSS
exceeds the resolved budget (RAM_GB/auto, or an explicit RSS_GUARD_GB
ceiling), free the least-used LRU expert slabs in place and lower ecap so the
cache cannot regrow. Slabs are >128 KB so glibc returns the pages to the
kernel immediately -- RSS actually drops.
Eviction never compacts the array: with PILOT_REAL the pilot worker holds
pointers into ecache[] across its preads, so the slot stays in place with
eid=-1/used=0 (first candidate for reuse); reserved slots (eid<0) are never
touched and victim selection happens under g_pilot_mx. resident_bytes is left
alone: LRU slots are never accounted there (only pin + dense).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CreateProcess cannot exec a shebang script, so the gateway exits during
setUpClass on the windows CI job. The gateway logic under test is
platform-independent and stays covered by the POSIX jobs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The gateway's tool-calling path had unit coverage (parse_tool_calls,
render_chat) but nothing exercised the real subprocess wire protocol or the
HTTP surface a coding client actually hits. #401 reports plain-text replies
where tool_calls were expected; every documented path checks out, so pin the
whole path down with a mock engine speaking SUBMIT/DATA/DONE and assert:
- non-stream: tool_calls populated, finish_reason tool_calls, no raw markers
- stream: markers suppressed across 20-way chunk splits, tool_calls delta
- tool-result round trip: <|observation|><tool_response> rendering, text reply
- no tools: plain text untouched
Also emit a stderr diagnosis when tools are declared and tool-call markers
are present in the reply but the strict parse matches nothing (typically
quantization-mangled output) pointing at COLI_TOOL_SALVAGE=1 -- the likely
field condition behind #401.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fix(test): make test_stops build on Windows (mkdtemp compat shim)
Conflict with #366's getenv_utf8 in compat.h resolved by keeping both blocks.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
monotophic