Re-derives the campaign's rtop8 breakthrough (fuse/rtop8-par @ b32439b,
commits 48b3a98 + b32439b) cleanly onto origin/dev @ 61004dc (post-#391
split), dropping the cb-chain commit (1130ac9) that branch was stacked
on. cb-chain touched only c/glm.c (now split into c/colibri.c); rtop8
touches only c/backend_metal.{h,mm} and c/tests/test_backend_metal.mm —
disjoint files, verified zero cb-chain remnants in this diff.
- r_top8_par: one SIMDGROUP per row (blocked lane ownership, taken
bitmask, shuffle-down argmax with ties->lower-index) replicating the
serial r_top8's first-max-wins ascending order exactly; topp/normk/
rscale tail verbatim on lane 0. Bench: serial 0.465 ms/layer (~55% of
the layer CB), parallel ~93x faster on the kernel, bit-exact.
- COLI_RTOP8 gate, default ON (renamed and inverted from the campaign's
COLI_RTOP8_PAR=0-default gate; COLI_RTOP8=0 is the opt-out escape).
- Expert-count generality (new hard requirement, REAP E=168 packages):
the parallel kernel's E<=256 contract is now enforced at EVERY call
site in host code (g_rtop8_width_ok / E<=256 checked before selecting
the pipeline), not just as an in-kernel defensive no-op -- closes a
latent gap where the campaign's SIMD-width guard only protected the
engine's automatic dispatch, not the standalone coli_metal_rtop8(par=1,
...) probe function metal-test itself uses. Out-of-contract requests
(E>256, or non-32-wide SIMD) now transparently run the serial kernel
instead of silently no-op'ing.
- metal-test: 16 new cases -- the campaign's 10-case exact-match fuzz
(now E-parametric) plus 6 new E-generality cases: E=168 (REAP) generic
+ massed-dup-ties, E=24 (<32 lane width) generic + ALL-EQUAL ties,
E=200 (a lane straddles the E boundary -- per=ceil(200/32)=7 doesn't
divide 200, so lane 28's ch[] block mixes 4 real indices with 3
sentinel ones; input is constructed to force those 4 into the top-8
deterministically, and the test asserts they were actually selected,
not just that serial==parallel), and E=257 (out-of-contract, proves
the auto-serial-fallback engages: this case is confirmed load-bearing
-- it fails without the new host-side guard). 15 stock + 10 ported +
6 new = 31 total metal-test cases.
Builds (METAL=0/1) and full C/python suites verified unchanged vs stock
61004dc (0 new warnings, identical pass counts). On-box A/B on this
branch is pending (see PR_BODY.md) -- no model runs performed here.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Under O_DIRECT the expert *weights* are read with the direct fd (bypassing the
page cache), so a POSIX_FADV_WILLNEED on them warms pages the demand read never
consumes -- pure wasted readahead on the disk, the scarcest resource when
streaming. The .qs scales are ALWAYS read buffered (pread on the normal fd), so
keep their WILLNEED. This only changes hint-only PILOT (PILOT=1, PILOT_REAL=0)
combined with DIRECT=1; fadvise is advisory, so output is bit-identical.
Prepared, not yet measured: needs a real NVMe + full model to A/B cold-stream
tok/s (PILOT=1 DIRECT=1, with/without this patch, TEMP=0). See #441.
pin_load computed npin from the RAM budget FIRST, then carved the
VRAM-ranked prefix out of those npin slots. With CUDA_RELEASE_HOST the
prefix's host slabs are freed right after upload — so on a multi-GPU
host the top-ranked experts consumed the RAM budget without occupying
RAM, and the CPU tier pinned only the leftovers. Measured on 6x RTX 5090
(251 GB): 9,280 VRAM + only 1,721 RAM pins (32.5 GB warm) on a box whose
RAM fits ~10k more — the cold tail then paid disk on every token and the
hit rate ceilinged at 99.0% forever.
Move the VRAM budget estimate above the npin finalization and make the
release-destined prefix ADDITIVE to the RAM-derived count. Same box,
same env plus the fix:
[PIN] placement: 9,280 VRAM + 10,176 RAM (192.5 GB warm)
expert hit rate 100.0% (pin 100.0% + lru 0.0%) — disk 0
1,024-token greedy decode: 3.62 -> 6.21 tok/s (+72%)
warm late segment (t=768-1024): 5.11 -> 5.73 tok/s (+12%)
prefill: 10.4 -> 8.8 s
The whole-run gain is the LRU warmup penalty disappearing (full
residency from the first token); the late-segment gain is the steady
~0.9%-miss disk residue recovered. Non-release configs are untouched:
prefix_est stays 0 and the arithmetic reduces to today's exactly.
The __AVXVNNI__ branches of dot_i8i8/dot_i4i8 (quant.h) accumulated every vpdpbusd
into a single register, a serial dependency chain (vpdpbusd is latency-bound ~5c).
Use 4 independent accumulators, mirroring the NEON 4-accumulator path in this file.
On x86 the tiled SMMLA _mm path is ARM-only, so these two dots are THE int8/int4
IDOT matmul kernels on x86 (g_idot=1 default), prefill and decode.
Integer accumulation is associative -> bit-identical (test_idot passes bit-for-bit).
Microbench (tests/bench_idot, i7-12700H P-core, I=6144):
dot_i8i8 6.85 -> 19.26 GB/s (2.81x)
dot_i4i8 3.41 -> 7.25 GB/s (2.13x)
Re-derivation of e7/disk-class-instr @ de6dd6d (base caa49f7) onto
origin/dev @ 61004dc, after PR #391 split c/glm.c into c/colibri.c plus
quant.h/sample.h/kv_persist.h/telemetry.h/grammar.h. Semantic equivalence,
not a copy: same events, same accounting, re-sited onto the new tree.
Placement: colibri.c, unchanged from before the split. telemetry.h (#391)
is the dashboard/stats module (HWINFO/TIERS/EMAP/HITS protocol lines +
usage persistence) — prof_report(), expert_load_impl(), moe(), g_prof_io
and g_edisk_ns all stayed in colibri.c, so DISK-CLASS's aggregation and
printing follow them there. No relocation needed, no DEVIATION.
Read path: #362 (prefetcher-v3, 22509fc) turned out to be testbed-only
scope per its own merge message ("glm.c untouched") — confirmed zero
c/glm.c changes in that merge's diff. The seven expert_load() call sites
this patch touches (pipe_worker, expert_host_ensure, moe()'s OMP miss
loop, pilot_realload, repin_pass_limit, pin_load x2) are structurally
identical to the pre-split tree; the demand flag re-attaches at the same
sites with the same semantics (1 only at moe()'s own PIPE/OMP miss path,
0 everywhere else), so DISK-CLASS still counts demand loads only.
One real drift, unrelated to #362: #417 (cfcc742) fixed the exact "Metal
pre-routed FASE A never bumps the real elast/eaccess_clock" defect this
feature's comments described as a documented, deliberately-unfixed
upstream issue — the real clock now ticks in FASE A too. The private
elast_dc/eaccess_clock_dc clock is kept anyway: its job was never only
to route around that freeze, it also snapshots pre-bump state so a
call's own routing bump can't contaminate its own classification, and
keeping DISK-CLASS's bookkeeping fully separate from stock elast state
is what makes "byte-identical with PROF=0" provable by construction
instead of by argument. Code comments referencing the old defect are
updated to reflect the fix (historical note + #417/cfcc742 pointer)
rather than describing a bug that no longer exists.
Gates: make glm METAL=0 and METAL=1 both clean, zero warnings (matches
stock 61004dc, also built clean with zero warnings for comparison).
make test-c: 0 failures. make test-python: 77 tests, OK.
Authored by Fable 5 in Claude Code, analysis in partnership with
@Monotophic.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
GLM-5.2's config defines three stop tokens: <|endoftext|>, <|user|>, and
<|observation|>. In serve mode, when the model generates <tool_call> blocks,
int4-quantized logit noise can cause argmax to pick a <|user|> or
<|observation|> token ID, immediately stopping generation.
The <|user|> and <|observation|> tokens are role markers handled by the
Python API server, not the C engine. Filter them out in stops_arm() when
SERVE=1, keeping only <|endoftext|> as the stop token.
- Add SERVE-mode guard in stops_arm() that retains only the EOS token
- Log the number of filtered tokens for diagnostics
The generation() method was re-extracting tools from the raw request body,
bypassing the filtering that render_chat() applies when tool_choice is a
function object (forced function call). This caused parse_tool_calls() to
receive the unfiltered tool list, producing incorrect or missing tool_calls
in the response.
- Add tools and tool_choice parameters to generation() signature
- Pass them from chat_completion() after render_chat() processing
- Add early structural validation of tools in generation_options()
for clear HTTP 400 errors on malformed input
The TC_W4A16 branch of coli_cuda_expert_group handled every expert in a
per-expert loop: rows >= threshold got the Tensor Core path, everything
below fell back to 4 naive launches per expert. At decode every expert
has 1 row, so the whole group rode the fallback — ~981 quant_matmul
micro-launches per token (#431's measured flood) — while the grouped
3-launch path (grouped_hidden_w4_dual + silu + grouped_down_w4) sat one
else-if below, unreachable whenever the PREFILL tuning flag was set.
Gate the branch on 'at least one expert reaches the TC row threshold':
all-small groups (decode) now fall through to the grouped kernels.
Measured on 6x RTX 5090 (full residency, 39 forwards under nsys):
expert-side quant_matmul instances drop 981 -> 337 per forward, total
launches ~1,490 -> ~850 per token (-43%). Wall-clock is parity at the
A/B operating point — the win is structural (PR-C graph node count,
launch-tax share at champion speed).
Behavioural fix folded in: before this change, toggling TC_W4A16 — a
prefill-only optimization — changed DECODE output text (kernel-family
divergence, #100 class). After it, decode always uses the grouped
family: TC_W4A16=1 and =0 now produce byte-identical decode text
(verified, 96-token greedy A/B), and the flag affects only the prefill
it was built for.
First increment of the #431 plan (device router -> indirect kernels ->
one-graph decode). At decode (S=1) on the pipe2 path, the router runs on
the layer's home device: a tiny E x D logits GEMV + sigmoid, then a
single-thread selection kernel that clones moe()'s plain routing path
verbatim — bias-augmented top-K by choice with strict-> tie-breaking,
weights from the raw logit, route-level TOPP truncation, norm_topk,
routed_scale. Results pack into one scratch buffer and come back in a
single ~68-byte D2H; moe() consumes them through the same pre-routed
shortcut the Metal layer-CB uses (g_pre_idx, #417 bookkeeping included),
so usage/heat/recency accounting is identical to the CPU router.
Structural value: routing becomes available ON the device timeline,
which is what PR-B (indirect expert kernels, static topology) and PR-C
(whole-decode CUDA Graph) build on.
Opt-in, default off. Gated to the plain routing path — CACHE_ROUTE,
ROUTE_P and ROUTE_TRACE keep the CPU ranking they need; any upload or
launch failure falls back to the CPU router silently. Router weights
(E x D f32, ~6.3 MB/layer) upload lazily to the layer's home device.
tests/test_router_cuda.cu: kernel-vs-CPU-reference oracle over 200
random trials (mixed TOPP/norm_topk/scale): 200/200 exact selections,
zero near-tie flips, zero weight mismatches on a 5090.
Update the rebased documentation and comments to name colibri.c after the #391 coordinator split, and correct the E5 comparison table to reflect its instrumentation and OOM-unwind touches.
Co-authored-by: Christopher Brand <brand.christopher.c@gmail.com>
1. DEFENSIVE (backend_metal.mm, coli_metal_register): re-registering a live
base overwrote s.buf and resset_add'ed the new wrapper without removing
the OLD one from the residency set -- ARC drops our reference but the set
retains the object and keeps its pages resident: a leak and a
set/g_slabs divergence. The found branch now stashes the replaced
wrapper under g_slab_mtx and, after dropping the lock (round-1 invariant:
no Metal call under the slab lock), resset_remove(old)s it before
resset_add(b); identical old==b early-outs both set operations.
Invariant defended, stated in the comment: residency-set membership
mirrors g_slabs exactly. No in-tree caller re-registers a live base
today (audited) -- closed defensively.
2. DOC (SUMMARY.md): the moe_submit lifecycle bullet still said
resset_flush commits "under g_slab_mtx" -- stale text from before the
round-1 mutex split; the code takes g_resset_mtx and never touches the
slab lock there. Parenthetical corrected; register bullet updated to
describe the re-register hygiene from item 1.
expert_load's fslab-OOM unwind freed s->slab via compat_aligned_free without
coli_metal_unregister -- a pre-existing gap on main/dev (validator-confirmed
during E4) that leaves a stale g_slabs entry over freed memory, letting
resolve() hand the GPU a dangling pointer. Under COLI_METAL_RESSET=1 the
exposure is strictly longer-lived: the wrapped buffer would stay a permanent
residency-set member over the freed pages instead of stock's transient
last-command-buffer window. Ported from e4/metal-heap validator fix 6753225,
adapted to dev's non-heap code shape (no coli_metal_heap_free wrapper --
plain unregister-before-free).
The uring_load_add analog (E4 audit round-2 insurance) is deliberately not
carried: that arm is #ifdef __linux__-gated while COLI_METAL is macOS-only,
so it is dead code on every real build target, and unlike E4 this branch has
no allocation-path reason to touch the function.
Reachable only through allocation failure mid-load; verified by inspection
and clean builds (no OOM-injection harness in tree). SUMMARY.md item 10
updated to "carried on this branch".
1. REQUIRED (backend_metal.mm): no Metal call runs under g_slab_mtx anymore.
The set mutations (addAllocation/removeAllocation/commit) and the dirty
flag moved to a dedicated g_resset_mtx; coli_metal_register/unregister do
their g_slabs bookkeeping under g_slab_mtx exactly as stock, then call
resset_add/resset_remove after dropping it (still before returning, so
unregister's remove+commit still lands before the caller frees the host
memory). The original shape -- commit under the slab lock the parallel OMP
loader threads contend on -- was structurally identical to the mutex-over-
live-Metal-call bug E4's audit round 2 fixed, the leading suspect for its
replicated +12s expert-disk regression. The register->flush->resolve
happens-before argument survives the split: resset_add completes inside
coli_metal_register before it returns, the engine cannot dispatch an
expert before its load returns, and add/flush are serialized by
g_resset_mtx (comment at resset_add). The two mutexes are never held
together, so no lock-order hazard exists.
2. REQUIRED (backend_metal.mm, backend_metal.h, glm.c): the resset_flush
cost in moe_submit -- deliberately outside the g_t_setup window so the
A/B harness counters keep their meaning -- was invisible. New
g_t_resset_flush accumulator timed around the flush, exported via
coli_metal_resset_stats(), printed by profile_print as a separate
gate-on-only "METAL-RESSET: flush" line (mirrors E4's METAL-HEAP line;
the METAL: line the harness parses keeps its exact format; stock output
byte-identical -- the stats call returns 0 with the gate off and the
whole block sits in the pre-existing #ifdef COLI_METAL arm). Register-
side add/remove costs land in the existing t_ewait window; comment at
resset_add says so instead of a second counter.
3. REQUIRED (SUMMARY.md; the moe_submit commit message was already reworded
in place, pre-push): corrected two false attributions -- the hazard-
tracking and thread-safety statements were credited to the SDK header,
which is silent on both; the actual source is Apple's online
MTLResidencySet class reference and residency-set adoption guide
(fetched 2026-07-18). Note: the scaffolding commit's message was checked
and contains no such claim, so it was left untouched.
4. DOC (SUMMARY.md): the pre-existing fslab OOM-unwind bug (glm.c ~1868-73,
frees s->slab without unregister) has a strictly longer-lived exposure
under E5 (permanent set member vs stock's transient per-CB declaration).
Out of scope here; the upstream PR built from E5 must carry the one-line
fix (reference: E4 branch commit 6753225).
The one seam the mechanism history actually implicates: moe_submit's `use`
vector (resolved expert weight/scale slabs) is the only useResource: loop
whose length scales with LRU cache size. With COLI_METAL_RESSET=1, skip that
loop entirely -- the queue-attached MTLResidencySet already guarantees those
buffers are resident -- after resset_flush() commits any pending adds from a
loader burst. Every other useResource: call site (bind_gemv's weight/scale
buffers, attn_decode/layer_decode's Lb/Rb/kvbW/kvbS/inB/pnB/rwB/rbB,
coli_metal_gemm's wb/sb) is left unconditionally unchanged: none of them
scale with cache size, and Lb/Rb carry real GPU-side write traffic ordered by
existing explicit memoryBarrierWithScope: calls, not by useResource:'s hazard
tracking -- narrowing the blast radius rather than removing useResource:
uniformly. Full reasoning, the hazard-tracking tradeoff (residency sets don't
support hazard tracking, per Apple's MTLResidencySet developer documentation
and residency-set adoption guide; the SDK header itself is silent on the
topic), and every judgment call in UNCERTAINTIES: see SUMMARY.md.
Gate off is unaffected: g_resset_enabled is false, so the useResource: loop
runs exactly as before.
Env-gated (COLI_METAL_RESSET=1) persistent MTLResidencySet attached to g_queue
(macOS 15+, @available-guarded with a one-line stderr fallback). Adds
resset_add/resset_remove/resset_flush helpers, wires them into the existing
coli_metal_register/coli_metal_unregister/coli_metal_shutdown bodies -- no new
functions in backend_metal.h, no glm.c changes, every existing call site keeps
its current signature and behavior.
register() defers the commit (resset_add just marks dirty, under the same
g_slab_mtx that already serializes parallel OMP loader threads) so a loader
burst doesn't pay a commit per slab; unregister() commits synchronously and
immediately, because the caller frees the underlying host memory right after
it returns and a deferred removal would leave the set referencing freed
memory. Nothing yet reads g_resset_enabled to change dispatch behavior --
this commit is bookkeeping only, gate on or off, so it does not change what
any command buffer does (verified by inspection: no useResource:/useHeap:
call site is touched here).
Gate off (default) is byte-for-byte the stock path: g_resset_enabled starts
false and nothing sets it outside the COLI_METAL_RESSET branch in
coli_metal_init, so every new helper is a no-op.
See SUMMARY.md (next commit) for the full design and UNCERTAINTIES.
The has_mtp completeness probe checked for `mlp.experts.255.down_proj.weight`,
which only exists when n_routed_experts == 256. REAP-pruned checkpoints (and any
MoE with a different expert count) have fewer experts, so the probe spuriously
reported has_mtp=0 and disabled MTP speculative decode even though the head was
present and complete. Probe `mlp.experts.<n_experts-1>` instead.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extend resource_plan to classify the hardware into bottleneck regimes
(disk / memory / mixed / compute) and derive tuning knobs automatically:
MTP: off when compute-bound (42% loss at full residency, #389)
or disk-bound with <90% hit (union growth adds reads)
PIPE: COLI_CUDA_PIPE=1 single-GPU, =2 multi-GPU, PIPE=1 CPU disk
NUMA: selective interleave for GPU hosts, blanket hint for CPU-only
PIN: PIN_GB=all when fully resident + no GPU
OMP: COLI_NO_OMP_TUNE=1 for Metal (spin steals GPU power)
`coli plan` now shows an auto-tune section with each knob and its
reason. `environment_for_plan()` applies them via setdefault so
explicit user settings always win.
plan version stays at 2 (additive fields: bottleneck_class,
projected_hit_rate, tune). 7 new tests covering all regimes.
Rename glm.c → colibri.c and extract four self-contained modules
into header-only files (same pattern as st.h/tier.h/grammar.h):
quant.h (672 lines) — SIMD matmul kernels, quantization
sample.h (143 lines) — RNG, top-p sampling, stop-set
kv_persist.h (121 lines) — .coli_kv disk persistence
telemetry.h (189 lines) — dashboard protocol, stats, usage
Main engine file shrinks from 6588 to 5396 lines (−18%).
Build system: primary target is now colibri$(EXE); `make glm`
remains as a phony alias for backward compat. CI, setup.sh,
coli CLI, and all 10 test files that include the engine are
updated. make check passes (C + Python, 73 tests, zero warnings).
On Metal, when routing is precomputed on the GPU (g_pre_idx), the moe fast
path bumps eusage/ehit/eheat for the selected experts but skips the one thing
the full CPU router does at the equivalent site: elast[layer][e] =
++eaccess_clock. So the session-local recency clock advances during prefill
(full router) but freezes the moment GPU-prerouted decode starts, and REPIN's
tier_pick_lfru() tie-breaker then runs on stale recency for the rest of the
run. Mirror the exact update the non-Metal path already does. Inside
#ifdef COLI_METAL, so CPU/CUDA are untouched; elast only feeds the LFRU
eviction heuristic, so this cannot affect output, only which experts REPIN
keeps warm.
Found and reported by @monotophic with a source-level trace repro
(ELAST_TRACE). Fix is inspection-verified against line ~3055; needs a
Metal build to exercise end-to-end.
Closes#417
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Colibri loads model directories and safetensors from mirrors it does not
control, so the file's declared shapes and byte spans are attacker-influenced
input. Three memory-safety holes on that boundary, independently confirmed
(incl. a from-scratch adversarial audit that re-derived the same two) and
present in the shipped v1.0.0:
- st.h st_read_f32: numel came from the shape, nbytes from the offsets, with
no cross-check. A crafted tensor whose shape inflates numel past nbytes made
the BF16/F16 loop read past the malloc'd raw buffer and the F32 memcpy write
past the caller's config-sized destination (heap OOB read + write). Now
enforce numel*esz == nbytes before any copy.
- st.h header parse: the shape product could overflow int64 to a small/negative
numel that would then pass the cross-check. Guard each multiply.
- glm.c qt_resolve_fmt (new, replaces the three duplicated "?1:?2:3" fmt sites
in qt_from_disk and both expert_load arms): the old inference SILENTLY fell
to int2 for any unrecognized weight byte count, so a too-short weight became
a valid int2 whose matmul read O*I nibbles past the buffer; and an oversized
scale array overflowed the per-row t->s. Now the weight bytes must match a
known int8/int4/int2 layout and the scale array must match the expected
per-row (O) or grouped (O*ng) cardinality, else refuse.
- glm.c config/generation_config slurp: unbounded ftell -> malloc(n+1) gave a
hostile file a load-time OOM, and on malloc failure b[got]=0 was a NULL
deref. Cap at 256 MB and NULL-check.
Verified: TF token-exactness unchanged on every quant format (full-precision
32/32, int4 11/32, int2 1/32, mix 5/32 -- byte-identical to the pre-change
binary); fmt=4 grouped path preserved (the scale check is by construction the
same condition detect_group_size already imposed); a hand-crafted hostile
safetensors is refused cleanly; ASan+UBSan clean on legit and hostile loads
(only the pre-existing intentional startup leaks remain).
These are the C trust-boundary items of #368, landed as a minimal standalone
fix; the server-side and build items of that PR follow via its rebase.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
cap_for_ram's projection is an estimate: on the GB10 (#403) long generations
overshot it by ~40 GB (projected 74.4, real 115.6) and the kernel killed the
engine three times. Run D of the issue proves a low cap CONTAINS the growth;
this guard does that automatically, keyed on MEASURED RSS instead of the
projection.
At the repin safe point (no moe in flight), every ~16 emitted tokens: if RSS
exceeds the resolved budget (RAM_GB/auto, or an explicit RSS_GUARD_GB
ceiling), free the least-used LRU expert slabs in place and lower ecap so the
cache cannot regrow. Slabs are >128 KB so glibc returns the pages to the
kernel immediately -- RSS actually drops.
Eviction never compacts the array: with PILOT_REAL the pilot worker holds
pointers into ecache[] across its preads, so the slot stays in place with
eid=-1/used=0 (first candidate for reuse); reserved slots (eid<0) are never
touched and victim selection happens under g_pilot_mx. resident_bytes is left
alone: LRU slots are never accounted there (only pin + dense).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CreateProcess cannot exec a shebang script, so the gateway exits during
setUpClass on the windows CI job. The gateway logic under test is
platform-independent and stays covered by the POSIX jobs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The gateway's tool-calling path had unit coverage (parse_tool_calls,
render_chat) but nothing exercised the real subprocess wire protocol or the
HTTP surface a coding client actually hits. #401 reports plain-text replies
where tool_calls were expected; every documented path checks out, so pin the
whole path down with a mock engine speaking SUBMIT/DATA/DONE and assert:
- non-stream: tool_calls populated, finish_reason tool_calls, no raw markers
- stream: markers suppressed across 20-way chunk splits, tool_calls delta
- tool-result round trip: <|observation|><tool_response> rendering, text reply
- no tools: plain text untouched
Also emit a stderr diagnosis when tools are declared and tool-call markers
are present in the reply but the strict parse matches nothing (typically
quantization-mangled output) pointing at COLI_TOOL_SALVAGE=1 -- the likely
field condition behind #401.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fix(test): make test_stops build on Windows (mkdtemp compat shim)
Conflict with #366's getenv_utf8 in compat.h resolved by keeping both blocks.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both coli's cuda_binary() and doctor.py's cuda_linkage() detect CUDA support
by running `ldd` on the engine binary and looking for a linked libcudart —
but that's Linux-only (cuda_binary() checks sys.platform != "linux", and
cuda_linkage() checks os.name != "posix", both short-circuiting to False on
win32). Windows CUDA_DLL=1 builds never link libcudart at all: glm.exe loads
coli_cuda.dll dynamically via LoadLibrary at startup (backend_loader.c), so
there's no import-table entry for ldd/dumpbin to find in the first place.
The practical effect: `coli doctor` always reported "NVIDIA GPU detected but
the engine is CPU-only" on Windows, and `coli run/chat/serve --gpu ...`
always hard-exited with "--gpu needs the CUDA build" — even on a correctly
built CUDA_DLL=1 binary with coli_cuda.dll sitting right next to glm.exe.
Fix: on win32, detect a COLI_CUDA build by scanning glm.exe for the marker
string "[CUDA] mode: routed experts", which only exists in code compiled
under #ifdef COLI_CUDA (see glm.c's cuda init block), then confirm
coli_cuda.dll actually sits next to the binary — mirroring the Linux
"linked but missing" distinction. Linux/macOS detection is unchanged.
Verified on Windows 11 with mingw-w64 GCC 16.1.0 + CUDA 13.2 + RTX 5080:
`coli doctor` now reports "CUDA engine and devices are available", and
`coli run --gpu 0 --vram 8` populates VRAM (confirmed via the engine's own
"[CUDA] resident set: N tensors, X GB VRAM" runtime log) instead of exiting.
- c/version.py: single source of truth (__version__ = "1.0.0")
- coli: reads version.py, banner shows dynamic version, --version flag
- .github/workflows/release.yml: tag push triggers cross-platform build
(Linux x86_64, macOS ARM64, Windows x86_64) and creates a GitHub
Release with packaged binaries + changelog notes
- CHANGELOG.md: v1.0.0 baseline documenting all shipped features
To cut a release:
1. bump c/version.py
2. add a CHANGELOG section
3. git tag v1.0.0 && git push --tags
Per-row int4 quantization of the MTP head's eh_proj [D,2D] zeroes its ENTIRE
embedding half: the tensor's two column halves differ in scale by ~20-30x per
row (embedding-half absmax ~0.05, hidden-half ~1.5 on GLM-5.2), so a single
per-row scale (absmax/7) puts every embedding-half weight below half a
quantization step and np.rint rounds it to exact zero (packed bytes 0x88).
The draft head then cannot see the input token and MTP acceptance collapses
to ~0% — the mechanism behind issue #8's measurement (int4: 0-4%; int8:
39-59%), and the reason --mtp already defaults to --ebits 8.
This showed up in the wild: a published pre-converted container had its MTP
shards made with an explicit --ebits 4 and drafts were pure garbage on every
backend (deterministic, data-side; verified byte-for-byte by reproducing the
published bytes with quant_int4 on the official BF16 rows).
- tools/repair_mtp_int8.py: repairs such a container IN PLACE — finds the MTP
layer's per-row-int4 dense tensors (eh_proj, q/kv/o projections, shared
experts; routed experts untouched), re-downloads only those from the FP8
source repo (~355 MB of HTTP range reads, no torch, no token), requantizes
at int8 with quant_int8's exact math, and rewrites the shards atomically
with *.bak-int4 backups. --dry-run to inspect. The engine picks up int8
automatically (qt_from_disk detects format by blob size).
Validated on GLM-5.2 744B: MTP acceptance 0% -> 100% (greedy), 1.11 -> 3.20
tokens/forward, decode 0.18 -> 0.30 tok/s (Metal, 4-bit KV).
- convert_fp8_to_int4.py: print a warning when --mtp is combined with
--ebits <8 and per-row scales (the exact footgun above); --group-size 128
remains a valid int4 alternative since group scales give the embedding
half its own scale.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two gaps in the local path, both raised in #383:
- The main --indir pass copied only config.json while the download path
copies four files; without tokenizer.json the converted container can't
run chat/serve. Copy the same four, print what was copied, and warn
when the source is missing any (tokenizer.json called out explicitly).
- Local passes restarted from shard 0 on every rerun. out-NNNNN names
count EMITTED shards, not input indexes (shards with no relevant
tensors emit nothing), so the download path's exists-check can't be
mirrored directly: a sidecar manifest per prefix records
input -> output (or empty) plus the conversion parameters, written
atomically after each shard. Rerun skips what matches and refuses a
parameter mismatch on the same outdir instead of mixing containers
(the #355 failure mode).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
coli run hardcoded the nothink template (<|assistant|><think></think>),
so THINK=1 had no effect in one-shot runs while serve mode honors it
(glm.c serve template picks <think> vs <think></think> from the same
env var). Pick the template the same way here.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Vincenzo Fornaro