Re-derives the campaign's rtop8 breakthrough (fuse/rtop8-par @ b32439b,
commits 48b3a98 + b32439b) cleanly onto origin/dev @ 61004dc (post-#391
split), dropping the cb-chain commit (1130ac9) that branch was stacked
on. cb-chain touched only c/glm.c (now split into c/colibri.c); rtop8
touches only c/backend_metal.{h,mm} and c/tests/test_backend_metal.mm —
disjoint files, verified zero cb-chain remnants in this diff.
- r_top8_par: one SIMDGROUP per row (blocked lane ownership, taken
bitmask, shuffle-down argmax with ties->lower-index) replicating the
serial r_top8's first-max-wins ascending order exactly; topp/normk/
rscale tail verbatim on lane 0. Bench: serial 0.465 ms/layer (~55% of
the layer CB), parallel ~93x faster on the kernel, bit-exact.
- COLI_RTOP8 gate, default ON (renamed and inverted from the campaign's
COLI_RTOP8_PAR=0-default gate; COLI_RTOP8=0 is the opt-out escape).
- Expert-count generality (new hard requirement, REAP E=168 packages):
the parallel kernel's E<=256 contract is now enforced at EVERY call
site in host code (g_rtop8_width_ok / E<=256 checked before selecting
the pipeline), not just as an in-kernel defensive no-op -- closes a
latent gap where the campaign's SIMD-width guard only protected the
engine's automatic dispatch, not the standalone coli_metal_rtop8(par=1,
...) probe function metal-test itself uses. Out-of-contract requests
(E>256, or non-32-wide SIMD) now transparently run the serial kernel
instead of silently no-op'ing.
- metal-test: 16 new cases -- the campaign's 10-case exact-match fuzz
(now E-parametric) plus 6 new E-generality cases: E=168 (REAP) generic
+ massed-dup-ties, E=24 (<32 lane width) generic + ALL-EQUAL ties,
E=200 (a lane straddles the E boundary -- per=ceil(200/32)=7 doesn't
divide 200, so lane 28's ch[] block mixes 4 real indices with 3
sentinel ones; input is constructed to force those 4 into the top-8
deterministically, and the test asserts they were actually selected,
not just that serial==parallel), and E=257 (out-of-contract, proves
the auto-serial-fallback engages: this case is confirmed load-bearing
-- it fails without the new host-side guard). 15 stock + 10 ported +
6 new = 31 total metal-test cases.
Builds (METAL=0/1) and full C/python suites verified unchanged vs stock
61004dc (0 new warnings, identical pass counts). On-box A/B on this
branch is pending (see PR_BODY.md) -- no model runs performed here.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Under O_DIRECT the expert *weights* are read with the direct fd (bypassing the
page cache), so a POSIX_FADV_WILLNEED on them warms pages the demand read never
consumes -- pure wasted readahead on the disk, the scarcest resource when
streaming. The .qs scales are ALWAYS read buffered (pread on the normal fd), so
keep their WILLNEED. This only changes hint-only PILOT (PILOT=1, PILOT_REAL=0)
combined with DIRECT=1; fadvise is advisory, so output is bit-identical.
Prepared, not yet measured: needs a real NVMe + full model to A/B cold-stream
tok/s (PILOT=1 DIRECT=1, with/without this patch, TEMP=0). See #441.
pin_load computed npin from the RAM budget FIRST, then carved the
VRAM-ranked prefix out of those npin slots. With CUDA_RELEASE_HOST the
prefix's host slabs are freed right after upload — so on a multi-GPU
host the top-ranked experts consumed the RAM budget without occupying
RAM, and the CPU tier pinned only the leftovers. Measured on 6x RTX 5090
(251 GB): 9,280 VRAM + only 1,721 RAM pins (32.5 GB warm) on a box whose
RAM fits ~10k more — the cold tail then paid disk on every token and the
hit rate ceilinged at 99.0% forever.
Move the VRAM budget estimate above the npin finalization and make the
release-destined prefix ADDITIVE to the RAM-derived count. Same box,
same env plus the fix:
[PIN] placement: 9,280 VRAM + 10,176 RAM (192.5 GB warm)
expert hit rate 100.0% (pin 100.0% + lru 0.0%) — disk 0
1,024-token greedy decode: 3.62 -> 6.21 tok/s (+72%)
warm late segment (t=768-1024): 5.11 -> 5.73 tok/s (+12%)
prefill: 10.4 -> 8.8 s
The whole-run gain is the LRU warmup penalty disappearing (full
residency from the first token); the late-segment gain is the steady
~0.9%-miss disk residue recovered. Non-release configs are untouched:
prefix_est stays 0 and the arithmetic reduces to today's exactly.
The __AVXVNNI__ branches of dot_i8i8/dot_i4i8 (quant.h) accumulated every vpdpbusd
into a single register, a serial dependency chain (vpdpbusd is latency-bound ~5c).
Use 4 independent accumulators, mirroring the NEON 4-accumulator path in this file.
On x86 the tiled SMMLA _mm path is ARM-only, so these two dots are THE int8/int4
IDOT matmul kernels on x86 (g_idot=1 default), prefill and decode.
Integer accumulation is associative -> bit-identical (test_idot passes bit-for-bit).
Microbench (tests/bench_idot, i7-12700H P-core, I=6144):
dot_i8i8 6.85 -> 19.26 GB/s (2.81x)
dot_i4i8 3.41 -> 7.25 GB/s (2.13x)
Re-derivation of e7/disk-class-instr @ de6dd6d (base caa49f7) onto
origin/dev @ 61004dc, after PR #391 split c/glm.c into c/colibri.c plus
quant.h/sample.h/kv_persist.h/telemetry.h/grammar.h. Semantic equivalence,
not a copy: same events, same accounting, re-sited onto the new tree.
Placement: colibri.c, unchanged from before the split. telemetry.h (#391)
is the dashboard/stats module (HWINFO/TIERS/EMAP/HITS protocol lines +
usage persistence) — prof_report(), expert_load_impl(), moe(), g_prof_io
and g_edisk_ns all stayed in colibri.c, so DISK-CLASS's aggregation and
printing follow them there. No relocation needed, no DEVIATION.
Read path: #362 (prefetcher-v3, 22509fc) turned out to be testbed-only
scope per its own merge message ("glm.c untouched") — confirmed zero
c/glm.c changes in that merge's diff. The seven expert_load() call sites
this patch touches (pipe_worker, expert_host_ensure, moe()'s OMP miss
loop, pilot_realload, repin_pass_limit, pin_load x2) are structurally
identical to the pre-split tree; the demand flag re-attaches at the same
sites with the same semantics (1 only at moe()'s own PIPE/OMP miss path,
0 everywhere else), so DISK-CLASS still counts demand loads only.
One real drift, unrelated to #362: #417 (cfcc742) fixed the exact "Metal
pre-routed FASE A never bumps the real elast/eaccess_clock" defect this
feature's comments described as a documented, deliberately-unfixed
upstream issue — the real clock now ticks in FASE A too. The private
elast_dc/eaccess_clock_dc clock is kept anyway: its job was never only
to route around that freeze, it also snapshots pre-bump state so a
call's own routing bump can't contaminate its own classification, and
keeping DISK-CLASS's bookkeeping fully separate from stock elast state
is what makes "byte-identical with PROF=0" provable by construction
instead of by argument. Code comments referencing the old defect are
updated to reflect the fix (historical note + #417/cfcc742 pointer)
rather than describing a bug that no longer exists.
Gates: make glm METAL=0 and METAL=1 both clean, zero warnings (matches
stock 61004dc, also built clean with zero warnings for comparison).
make test-c: 0 failures. make test-python: 77 tests, OK.
Authored by Fable 5 in Claude Code, analysis in partnership with
@Monotophic.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
GLM-5.2's config defines three stop tokens: <|endoftext|>, <|user|>, and
<|observation|>. In serve mode, when the model generates <tool_call> blocks,
int4-quantized logit noise can cause argmax to pick a <|user|> or
<|observation|> token ID, immediately stopping generation.
The <|user|> and <|observation|> tokens are role markers handled by the
Python API server, not the C engine. Filter them out in stops_arm() when
SERVE=1, keeping only <|endoftext|> as the stop token.
- Add SERVE-mode guard in stops_arm() that retains only the EOS token
- Log the number of filtered tokens for diagnostics
The generation() method was re-extracting tools from the raw request body,
bypassing the filtering that render_chat() applies when tool_choice is a
function object (forced function call). This caused parse_tool_calls() to
receive the unfiltered tool list, producing incorrect or missing tool_calls
in the response.
- Add tools and tool_choice parameters to generation() signature
- Pass them from chat_completion() after render_chat() processing
- Add early structural validation of tools in generation_options()
for clear HTTP 400 errors on malformed input
The TC_W4A16 branch of coli_cuda_expert_group handled every expert in a
per-expert loop: rows >= threshold got the Tensor Core path, everything
below fell back to 4 naive launches per expert. At decode every expert
has 1 row, so the whole group rode the fallback — ~981 quant_matmul
micro-launches per token (#431's measured flood) — while the grouped
3-launch path (grouped_hidden_w4_dual + silu + grouped_down_w4) sat one
else-if below, unreachable whenever the PREFILL tuning flag was set.
Gate the branch on 'at least one expert reaches the TC row threshold':
all-small groups (decode) now fall through to the grouped kernels.
Measured on 6x RTX 5090 (full residency, 39 forwards under nsys):
expert-side quant_matmul instances drop 981 -> 337 per forward, total
launches ~1,490 -> ~850 per token (-43%). Wall-clock is parity at the
A/B operating point — the win is structural (PR-C graph node count,
launch-tax share at champion speed).
Behavioural fix folded in: before this change, toggling TC_W4A16 — a
prefill-only optimization — changed DECODE output text (kernel-family
divergence, #100 class). After it, decode always uses the grouped
family: TC_W4A16=1 and =0 now produce byte-identical decode text
(verified, 96-token greedy A/B), and the flag affects only the prefill
it was built for.
First increment of the #431 plan (device router -> indirect kernels ->
one-graph decode). At decode (S=1) on the pipe2 path, the router runs on
the layer's home device: a tiny E x D logits GEMV + sigmoid, then a
single-thread selection kernel that clones moe()'s plain routing path
verbatim — bias-augmented top-K by choice with strict-> tie-breaking,
weights from the raw logit, route-level TOPP truncation, norm_topk,
routed_scale. Results pack into one scratch buffer and come back in a
single ~68-byte D2H; moe() consumes them through the same pre-routed
shortcut the Metal layer-CB uses (g_pre_idx, #417 bookkeeping included),
so usage/heat/recency accounting is identical to the CPU router.
Structural value: routing becomes available ON the device timeline,
which is what PR-B (indirect expert kernels, static topology) and PR-C
(whole-decode CUDA Graph) build on.
Opt-in, default off. Gated to the plain routing path — CACHE_ROUTE,
ROUTE_P and ROUTE_TRACE keep the CPU ranking they need; any upload or
launch failure falls back to the CPU router silently. Router weights
(E x D f32, ~6.3 MB/layer) upload lazily to the layer's home device.
tests/test_router_cuda.cu: kernel-vs-CPU-reference oracle over 200
random trials (mixed TOPP/norm_topk/scale): 200/200 exact selections,
zero near-tie flips, zero weight mismatches on a 5090.
Record that the explicit Objective-C++ warning build reports only the pre-existing unused TG constant inherited from current dev.
Co-authored-by: Christopher Brand <brand.christopher.c@gmail.com>
Update the rebased documentation and comments to name colibri.c after the #391 coordinator split, and correct the E5 comparison table to reflect its instrumentation and OOM-unwind touches.
Co-authored-by: Christopher Brand <brand.christopher.c@gmail.com>
1. DEFENSIVE (backend_metal.mm, coli_metal_register): re-registering a live
base overwrote s.buf and resset_add'ed the new wrapper without removing
the OLD one from the residency set -- ARC drops our reference but the set
retains the object and keeps its pages resident: a leak and a
set/g_slabs divergence. The found branch now stashes the replaced
wrapper under g_slab_mtx and, after dropping the lock (round-1 invariant:
no Metal call under the slab lock), resset_remove(old)s it before
resset_add(b); identical old==b early-outs both set operations.
Invariant defended, stated in the comment: residency-set membership
mirrors g_slabs exactly. No in-tree caller re-registers a live base
today (audited) -- closed defensively.
2. DOC (SUMMARY.md): the moe_submit lifecycle bullet still said
resset_flush commits "under g_slab_mtx" -- stale text from before the
round-1 mutex split; the code takes g_resset_mtx and never touches the
slab lock there. Parenthetical corrected; register bullet updated to
describe the re-register hygiene from item 1.
expert_load's fslab-OOM unwind freed s->slab via compat_aligned_free without
coli_metal_unregister -- a pre-existing gap on main/dev (validator-confirmed
during E4) that leaves a stale g_slabs entry over freed memory, letting
resolve() hand the GPU a dangling pointer. Under COLI_METAL_RESSET=1 the
exposure is strictly longer-lived: the wrapped buffer would stay a permanent
residency-set member over the freed pages instead of stock's transient
last-command-buffer window. Ported from e4/metal-heap validator fix 6753225,
adapted to dev's non-heap code shape (no coli_metal_heap_free wrapper --
plain unregister-before-free).
The uring_load_add analog (E4 audit round-2 insurance) is deliberately not
carried: that arm is #ifdef __linux__-gated while COLI_METAL is macOS-only,
so it is dead code on every real build target, and unlike E4 this branch has
no allocation-path reason to touch the function.
Reachable only through allocation failure mid-load; verified by inspection
and clean builds (no OOM-injection harness in tree). SUMMARY.md item 10
updated to "carried on this branch".
1. REQUIRED (backend_metal.mm): no Metal call runs under g_slab_mtx anymore.
The set mutations (addAllocation/removeAllocation/commit) and the dirty
flag moved to a dedicated g_resset_mtx; coli_metal_register/unregister do
their g_slabs bookkeeping under g_slab_mtx exactly as stock, then call
resset_add/resset_remove after dropping it (still before returning, so
unregister's remove+commit still lands before the caller frees the host
memory). The original shape -- commit under the slab lock the parallel OMP
loader threads contend on -- was structurally identical to the mutex-over-
live-Metal-call bug E4's audit round 2 fixed, the leading suspect for its
replicated +12s expert-disk regression. The register->flush->resolve
happens-before argument survives the split: resset_add completes inside
coli_metal_register before it returns, the engine cannot dispatch an
expert before its load returns, and add/flush are serialized by
g_resset_mtx (comment at resset_add). The two mutexes are never held
together, so no lock-order hazard exists.
2. REQUIRED (backend_metal.mm, backend_metal.h, glm.c): the resset_flush
cost in moe_submit -- deliberately outside the g_t_setup window so the
A/B harness counters keep their meaning -- was invisible. New
g_t_resset_flush accumulator timed around the flush, exported via
coli_metal_resset_stats(), printed by profile_print as a separate
gate-on-only "METAL-RESSET: flush" line (mirrors E4's METAL-HEAP line;
the METAL: line the harness parses keeps its exact format; stock output
byte-identical -- the stats call returns 0 with the gate off and the
whole block sits in the pre-existing #ifdef COLI_METAL arm). Register-
side add/remove costs land in the existing t_ewait window; comment at
resset_add says so instead of a second counter.
3. REQUIRED (SUMMARY.md; the moe_submit commit message was already reworded
in place, pre-push): corrected two false attributions -- the hazard-
tracking and thread-safety statements were credited to the SDK header,
which is silent on both; the actual source is Apple's online
MTLResidencySet class reference and residency-set adoption guide
(fetched 2026-07-18). Note: the scaffolding commit's message was checked
and contains no such claim, so it was left untouched.
4. DOC (SUMMARY.md): the pre-existing fslab OOM-unwind bug (glm.c ~1868-73,
frees s->slab without unregister) has a strictly longer-lived exposure
under E5 (permanent set member vs stock's transient per-CB declaration).
Out of scope here; the upstream PR built from E5 must carry the one-line
fix (reference: E4 branch commit 6753225).
The one seam the mechanism history actually implicates: moe_submit's `use`
vector (resolved expert weight/scale slabs) is the only useResource: loop
whose length scales with LRU cache size. With COLI_METAL_RESSET=1, skip that
loop entirely -- the queue-attached MTLResidencySet already guarantees those
buffers are resident -- after resset_flush() commits any pending adds from a
loader burst. Every other useResource: call site (bind_gemv's weight/scale
buffers, attn_decode/layer_decode's Lb/Rb/kvbW/kvbS/inB/pnB/rwB/rbB,
coli_metal_gemm's wb/sb) is left unconditionally unchanged: none of them
scale with cache size, and Lb/Rb carry real GPU-side write traffic ordered by
existing explicit memoryBarrierWithScope: calls, not by useResource:'s hazard
tracking -- narrowing the blast radius rather than removing useResource:
uniformly. Full reasoning, the hazard-tracking tradeoff (residency sets don't
support hazard tracking, per Apple's MTLResidencySet developer documentation
and residency-set adoption guide; the SDK header itself is silent on the
topic), and every judgment call in UNCERTAINTIES: see SUMMARY.md.
Gate off is unaffected: g_resset_enabled is false, so the useResource: loop
runs exactly as before.
Env-gated (COLI_METAL_RESSET=1) persistent MTLResidencySet attached to g_queue
(macOS 15+, @available-guarded with a one-line stderr fallback). Adds
resset_add/resset_remove/resset_flush helpers, wires them into the existing
coli_metal_register/coli_metal_unregister/coli_metal_shutdown bodies -- no new
functions in backend_metal.h, no glm.c changes, every existing call site keeps
its current signature and behavior.
register() defers the commit (resset_add just marks dirty, under the same
g_slab_mtx that already serializes parallel OMP loader threads) so a loader
burst doesn't pay a commit per slab; unregister() commits synchronously and
immediately, because the caller frees the underlying host memory right after
it returns and a deferred removal would leave the set referencing freed
memory. Nothing yet reads g_resset_enabled to change dispatch behavior --
this commit is bookkeeping only, gate on or off, so it does not change what
any command buffer does (verified by inspection: no useResource:/useHeap:
call site is touched here).
Gate off (default) is byte-for-byte the stock path: g_resset_enabled starts
false and nothing sets it outside the COLI_METAL_RESSET branch in
coli_metal_init, so every new helper is a no-op.
See SUMMARY.md (next commit) for the full design and UNCERTAINTIES.
- The vision: open the model up — run it, study it, improve it
- The idea: explain the core algorithm as a JIT for weights — parameters
as data staged across a heterogeneous hierarchy, learned from routing
- What's next: active placement/scheduling research; Kimi K2, Qwen3 MoE,
MiniMax on the model roadmap
- Acknowledgements: Z.ai, Moonshot AI, Alibaba Qwen, MiniMax, Allen AI
- hero: the measured expert atlas as a full-screen slowly-turning backdrop
- atlas rebuilt on real data: canonical atlas v1 (721 canonical + 637
gate-sensitive specialists, 10 measured categories) embedded inline;
position IS the measured affinity vector, colour = top topic
- demo: third panel 'the atlas, live' — token routing flashes measured
specialists for the active topic in both the brain grid and the galaxy;
added SQL and Chinese-poetry turns so cluster shifts are visible
- profiles/ladder updated to current community numbers (#82 NUMA 9.0-9.2,
#389 Xeon 1TB 5.42, #387 M5 Max 2.0, #161 GB10 3.33, #120 1.23);
unpublished TTFT/hit values shown as em-dash, never invented
- new sections: vision manifesto (run/study/improve), 'A JIT, but for
weights' algorithm explainer with tier stack, models roadmap
(Kimi K2 / Qwen3 MoE / MiniMax planned) + open-weights acknowledgements,
contribute cards
- visual pass: numbered sections, gradient type, glass panels, fixed nav
A zero-build static site under site/, deployed to GitHub Pages by Actions:
- hero with the pixel hummingbird, key numbers, CTA
- 'watch it think': a chat replay paced at measured decode speeds
(6x5090 / 128GB CPU / 5070 Ti / 25GB floor), with a live tok/s meter
and the full 19,456-expert grid — colour = tier, brightness = heat,
routed experts flash white per token
- the expert atlas as a draggable 3-D galaxy (measured-affinity clusters)
- three-tier explainer and the measured hardware ladder
- single HTML file, no dependencies, no build step; custom domain later
is just a site/CNAME + DNS
The has_mtp completeness probe checked for `mlp.experts.255.down_proj.weight`,
which only exists when n_routed_experts == 256. REAP-pruned checkpoints (and any
MoE with a different expert count) have fewer experts, so the probe spuriously
reported has_mtp=0 and disabled MTP speculative decode even though the head was
present and complete. Probe `mlp.experts.<n_experts-1>` instead.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extend resource_plan to classify the hardware into bottleneck regimes
(disk / memory / mixed / compute) and derive tuning knobs automatically:
MTP: off when compute-bound (42% loss at full residency, #389)
or disk-bound with <90% hit (union growth adds reads)
PIPE: COLI_CUDA_PIPE=1 single-GPU, =2 multi-GPU, PIPE=1 CPU disk
NUMA: selective interleave for GPU hosts, blanket hint for CPU-only
PIN: PIN_GB=all when fully resident + no GPU
OMP: COLI_NO_OMP_TUNE=1 for Metal (spin steals GPU power)
`coli plan` now shows an auto-tune section with each knob and its
reason. `environment_for_plan()` applies them via setdefault so
explicit user settings always win.
plan version stays at 2 (additive fields: bottleneck_class,
projected_hit_rate, tune). 7 new tests covering all regimes.
Lightweight i18n without react-i18next: a LocaleProvider context +
useLocale() hook with {{var}} interpolation, auto-detect from
navigator.language, persisted to localStorage.
98 translation keys across 4 locale files (en/zh-CN/zh-TW/it).
All user-visible strings in App/Brain/Profiling/ErrorBoundary are
now t() calls. Language switcher added to the sidebar footer.
Build clean (tsc + vite), 18 tests pass.
New files:
README.zh-CN.md — simplified Chinese (大陆用词)
README.it.md — Italian (the project's "mother tongue")
All four READMEs now link to each other in a consistent nav bar.
Updated zh-TW to reflect glm.c → colibri.c rename and new headers.
Rename glm.c → colibri.c and extract four self-contained modules
into header-only files (same pattern as st.h/tier.h/grammar.h):
quant.h (672 lines) — SIMD matmul kernels, quantization
sample.h (143 lines) — RNG, top-p sampling, stop-set
kv_persist.h (121 lines) — .coli_kv disk persistence
telemetry.h (189 lines) — dashboard protocol, stats, usage
Main engine file shrinks from 6588 to 5396 lines (−18%).
Build system: primary target is now colibri$(EXE); `make glm`
remains as a phony alias for backward compat. CI, setup.sh,
coli CLI, and all 10 test files that include the engine are
updated. make check passes (C + Python, 73 tests, zero warnings).
On Metal, when routing is precomputed on the GPU (g_pre_idx), the moe fast
path bumps eusage/ehit/eheat for the selected experts but skips the one thing
the full CPU router does at the equivalent site: elast[layer][e] =
++eaccess_clock. So the session-local recency clock advances during prefill
(full router) but freezes the moment GPU-prerouted decode starts, and REPIN's
tier_pick_lfru() tie-breaker then runs on stale recency for the rest of the
run. Mirror the exact update the non-Metal path already does. Inside
#ifdef COLI_METAL, so CPU/CUDA are untouched; elast only feeds the LFRU
eviction heuristic, so this cannot affect output, only which experts REPIN
keeps warm.
Found and reported by @monotophic with a source-level trace repro
(ELAST_TRACE). Fix is inspection-verified against line ~3055; needs a
Metal build to exercise end-to-end.
Closes#417
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds docs/quickstart.md — a step-by-step, no-experience-assumed walkthrough
from installing the build tools to the first coli chat, with per-OS
copy-paste commands (Ubuntu apt, Windows MSYS2 or prebuilt binary, macOS
brew), the ready-made HF int4 container plus the self-convert path, and an
honest 'what to expect' on disk-bound speed. Commands verified against
setup.sh and the coli subcommands; every cross-linked doc exists. Linked
from the README's Get started section.
Closes#414
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Vincenzo Fornaro