Re-derivation of e7/disk-class-instr @ de6dd6d (base caa49f7) onto
origin/dev @ 61004dc, after PR #391 split c/glm.c into c/colibri.c plus
quant.h/sample.h/kv_persist.h/telemetry.h/grammar.h. Semantic equivalence,
not a copy: same events, same accounting, re-sited onto the new tree.
Placement: colibri.c, unchanged from before the split. telemetry.h (#391)
is the dashboard/stats module (HWINFO/TIERS/EMAP/HITS protocol lines +
usage persistence) — prof_report(), expert_load_impl(), moe(), g_prof_io
and g_edisk_ns all stayed in colibri.c, so DISK-CLASS's aggregation and
printing follow them there. No relocation needed, no DEVIATION.
Read path: #362 (prefetcher-v3, 22509fc) turned out to be testbed-only
scope per its own merge message ("glm.c untouched") — confirmed zero
c/glm.c changes in that merge's diff. The seven expert_load() call sites
this patch touches (pipe_worker, expert_host_ensure, moe()'s OMP miss
loop, pilot_realload, repin_pass_limit, pin_load x2) are structurally
identical to the pre-split tree; the demand flag re-attaches at the same
sites with the same semantics (1 only at moe()'s own PIPE/OMP miss path,
0 everywhere else), so DISK-CLASS still counts demand loads only.
One real drift, unrelated to #362: #417 (cfcc742) fixed the exact "Metal
pre-routed FASE A never bumps the real elast/eaccess_clock" defect this
feature's comments described as a documented, deliberately-unfixed
upstream issue — the real clock now ticks in FASE A too. The private
elast_dc/eaccess_clock_dc clock is kept anyway: its job was never only
to route around that freeze, it also snapshots pre-bump state so a
call's own routing bump can't contaminate its own classification, and
keeping DISK-CLASS's bookkeeping fully separate from stock elast state
is what makes "byte-identical with PROF=0" provable by construction
instead of by argument. Code comments referencing the old defect are
updated to reflect the fix (historical note + #417/cfcc742 pointer)
rather than describing a bug that no longer exists.
Gates: make glm METAL=0 and METAL=1 both clean, zero warnings (matches
stock 61004dc, also built clean with zero warnings for comparison).
make test-c: 0 failures. make test-python: 77 tests, OK.
Authored by Fable 5 in Claude Code, analysis in partnership with
@Monotophic.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
GLM-5.2's config defines three stop tokens: <|endoftext|>, <|user|>, and
<|observation|>. In serve mode, when the model generates <tool_call> blocks,
int4-quantized logit noise can cause argmax to pick a <|user|> or
<|observation|> token ID, immediately stopping generation.
The <|user|> and <|observation|> tokens are role markers handled by the
Python API server, not the C engine. Filter them out in stops_arm() when
SERVE=1, keeping only <|endoftext|> as the stop token.
- Add SERVE-mode guard in stops_arm() that retains only the EOS token
- Log the number of filtered tokens for diagnostics
The generation() method was re-extracting tools from the raw request body,
bypassing the filtering that render_chat() applies when tool_choice is a
function object (forced function call). This caused parse_tool_calls() to
receive the unfiltered tool list, producing incorrect or missing tool_calls
in the response.
- Add tools and tool_choice parameters to generation() signature
- Pass them from chat_completion() after render_chat() processing
- Add early structural validation of tools in generation_options()
for clear HTTP 400 errors on malformed input
The TC_W4A16 branch of coli_cuda_expert_group handled every expert in a
per-expert loop: rows >= threshold got the Tensor Core path, everything
below fell back to 4 naive launches per expert. At decode every expert
has 1 row, so the whole group rode the fallback — ~981 quant_matmul
micro-launches per token (#431's measured flood) — while the grouped
3-launch path (grouped_hidden_w4_dual + silu + grouped_down_w4) sat one
else-if below, unreachable whenever the PREFILL tuning flag was set.
Gate the branch on 'at least one expert reaches the TC row threshold':
all-small groups (decode) now fall through to the grouped kernels.
Measured on 6x RTX 5090 (full residency, 39 forwards under nsys):
expert-side quant_matmul instances drop 981 -> 337 per forward, total
launches ~1,490 -> ~850 per token (-43%). Wall-clock is parity at the
A/B operating point — the win is structural (PR-C graph node count,
launch-tax share at champion speed).
Behavioural fix folded in: before this change, toggling TC_W4A16 — a
prefill-only optimization — changed DECODE output text (kernel-family
divergence, #100 class). After it, decode always uses the grouped
family: TC_W4A16=1 and =0 now produce byte-identical decode text
(verified, 96-token greedy A/B), and the flag affects only the prefill
it was built for.
First increment of the #431 plan (device router -> indirect kernels ->
one-graph decode). At decode (S=1) on the pipe2 path, the router runs on
the layer's home device: a tiny E x D logits GEMV + sigmoid, then a
single-thread selection kernel that clones moe()'s plain routing path
verbatim — bias-augmented top-K by choice with strict-> tie-breaking,
weights from the raw logit, route-level TOPP truncation, norm_topk,
routed_scale. Results pack into one scratch buffer and come back in a
single ~68-byte D2H; moe() consumes them through the same pre-routed
shortcut the Metal layer-CB uses (g_pre_idx, #417 bookkeeping included),
so usage/heat/recency accounting is identical to the CPU router.
Structural value: routing becomes available ON the device timeline,
which is what PR-B (indirect expert kernels, static topology) and PR-C
(whole-decode CUDA Graph) build on.
Opt-in, default off. Gated to the plain routing path — CACHE_ROUTE,
ROUTE_P and ROUTE_TRACE keep the CPU ranking they need; any upload or
launch failure falls back to the CPU router silently. Router weights
(E x D f32, ~6.3 MB/layer) upload lazily to the layer's home device.
tests/test_router_cuda.cu: kernel-vs-CPU-reference oracle over 200
random trials (mixed TOPP/norm_topk/scale): 200/200 exact selections,
zero near-tie flips, zero weight mismatches on a 5090.
- The vision: open the model up — run it, study it, improve it
- The idea: explain the core algorithm as a JIT for weights — parameters
as data staged across a heterogeneous hierarchy, learned from routing
- What's next: active placement/scheduling research; Kimi K2, Qwen3 MoE,
MiniMax on the model roadmap
- Acknowledgements: Z.ai, Moonshot AI, Alibaba Qwen, MiniMax, Allen AI
- hero: the measured expert atlas as a full-screen slowly-turning backdrop
- atlas rebuilt on real data: canonical atlas v1 (721 canonical + 637
gate-sensitive specialists, 10 measured categories) embedded inline;
position IS the measured affinity vector, colour = top topic
- demo: third panel 'the atlas, live' — token routing flashes measured
specialists for the active topic in both the brain grid and the galaxy;
added SQL and Chinese-poetry turns so cluster shifts are visible
- profiles/ladder updated to current community numbers (#82 NUMA 9.0-9.2,
#389 Xeon 1TB 5.42, #387 M5 Max 2.0, #161 GB10 3.33, #120 1.23);
unpublished TTFT/hit values shown as em-dash, never invented
- new sections: vision manifesto (run/study/improve), 'A JIT, but for
weights' algorithm explainer with tier stack, models roadmap
(Kimi K2 / Qwen3 MoE / MiniMax planned) + open-weights acknowledgements,
contribute cards
- visual pass: numbered sections, gradient type, glass panels, fixed nav
A zero-build static site under site/, deployed to GitHub Pages by Actions:
- hero with the pixel hummingbird, key numbers, CTA
- 'watch it think': a chat replay paced at measured decode speeds
(6x5090 / 128GB CPU / 5070 Ti / 25GB floor), with a live tok/s meter
and the full 19,456-expert grid — colour = tier, brightness = heat,
routed experts flash white per token
- the expert atlas as a draggable 3-D galaxy (measured-affinity clusters)
- three-tier explainer and the measured hardware ladder
- single HTML file, no dependencies, no build step; custom domain later
is just a site/CNAME + DNS
The has_mtp completeness probe checked for `mlp.experts.255.down_proj.weight`,
which only exists when n_routed_experts == 256. REAP-pruned checkpoints (and any
MoE with a different expert count) have fewer experts, so the probe spuriously
reported has_mtp=0 and disabled MTP speculative decode even though the head was
present and complete. Probe `mlp.experts.<n_experts-1>` instead.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extend resource_plan to classify the hardware into bottleneck regimes
(disk / memory / mixed / compute) and derive tuning knobs automatically:
MTP: off when compute-bound (42% loss at full residency, #389)
or disk-bound with <90% hit (union growth adds reads)
PIPE: COLI_CUDA_PIPE=1 single-GPU, =2 multi-GPU, PIPE=1 CPU disk
NUMA: selective interleave for GPU hosts, blanket hint for CPU-only
PIN: PIN_GB=all when fully resident + no GPU
OMP: COLI_NO_OMP_TUNE=1 for Metal (spin steals GPU power)
`coli plan` now shows an auto-tune section with each knob and its
reason. `environment_for_plan()` applies them via setdefault so
explicit user settings always win.
plan version stays at 2 (additive fields: bottleneck_class,
projected_hit_rate, tune). 7 new tests covering all regimes.
Lightweight i18n without react-i18next: a LocaleProvider context +
useLocale() hook with {{var}} interpolation, auto-detect from
navigator.language, persisted to localStorage.
98 translation keys across 4 locale files (en/zh-CN/zh-TW/it).
All user-visible strings in App/Brain/Profiling/ErrorBoundary are
now t() calls. Language switcher added to the sidebar footer.
Build clean (tsc + vite), 18 tests pass.
New files:
README.zh-CN.md — simplified Chinese (大陆用词)
README.it.md — Italian (the project's "mother tongue")
All four READMEs now link to each other in a consistent nav bar.
Updated zh-TW to reflect glm.c → colibri.c rename and new headers.
Rename glm.c → colibri.c and extract four self-contained modules
into header-only files (same pattern as st.h/tier.h/grammar.h):
quant.h (672 lines) — SIMD matmul kernels, quantization
sample.h (143 lines) — RNG, top-p sampling, stop-set
kv_persist.h (121 lines) — .coli_kv disk persistence
telemetry.h (189 lines) — dashboard protocol, stats, usage
Main engine file shrinks from 6588 to 5396 lines (−18%).
Build system: primary target is now colibri$(EXE); `make glm`
remains as a phony alias for backward compat. CI, setup.sh,
coli CLI, and all 10 test files that include the engine are
updated. make check passes (C + Python, 73 tests, zero warnings).
On Metal, when routing is precomputed on the GPU (g_pre_idx), the moe fast
path bumps eusage/ehit/eheat for the selected experts but skips the one thing
the full CPU router does at the equivalent site: elast[layer][e] =
++eaccess_clock. So the session-local recency clock advances during prefill
(full router) but freezes the moment GPU-prerouted decode starts, and REPIN's
tier_pick_lfru() tie-breaker then runs on stale recency for the rest of the
run. Mirror the exact update the non-Metal path already does. Inside
#ifdef COLI_METAL, so CPU/CUDA are untouched; elast only feeds the LFRU
eviction heuristic, so this cannot affect output, only which experts REPIN
keeps warm.
Found and reported by @monotophic with a source-level trace repro
(ELAST_TRACE). Fix is inspection-verified against line ~3055; needs a
Metal build to exercise end-to-end.
Closes#417
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds docs/quickstart.md — a step-by-step, no-experience-assumed walkthrough
from installing the build tools to the first coli chat, with per-OS
copy-paste commands (Ubuntu apt, Windows MSYS2 or prebuilt binary, macOS
brew), the ready-made HF int4 container plus the self-convert path, and an
honest 'what to expect' on disk-bound speed. Commands verified against
setup.sh and the coli subcommands; every cross-linked doc exists. Linked
from the README's Get started section.
Closes#414
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This sets the formatter to alejandra. I was trying to stay as close as
the original as I could; otherwise we could set it to `nixfmt` which
would keep list spacing more similar, but would introduce additional
indentation in a few places.
Colibri loads model directories and safetensors from mirrors it does not
control, so the file's declared shapes and byte spans are attacker-influenced
input. Three memory-safety holes on that boundary, independently confirmed
(incl. a from-scratch adversarial audit that re-derived the same two) and
present in the shipped v1.0.0:
- st.h st_read_f32: numel came from the shape, nbytes from the offsets, with
no cross-check. A crafted tensor whose shape inflates numel past nbytes made
the BF16/F16 loop read past the malloc'd raw buffer and the F32 memcpy write
past the caller's config-sized destination (heap OOB read + write). Now
enforce numel*esz == nbytes before any copy.
- st.h header parse: the shape product could overflow int64 to a small/negative
numel that would then pass the cross-check. Guard each multiply.
- glm.c qt_resolve_fmt (new, replaces the three duplicated "?1:?2:3" fmt sites
in qt_from_disk and both expert_load arms): the old inference SILENTLY fell
to int2 for any unrecognized weight byte count, so a too-short weight became
a valid int2 whose matmul read O*I nibbles past the buffer; and an oversized
scale array overflowed the per-row t->s. Now the weight bytes must match a
known int8/int4/int2 layout and the scale array must match the expected
per-row (O) or grouped (O*ng) cardinality, else refuse.
- glm.c config/generation_config slurp: unbounded ftell -> malloc(n+1) gave a
hostile file a load-time OOM, and on malloc failure b[got]=0 was a NULL
deref. Cap at 256 MB and NULL-check.
Verified: TF token-exactness unchanged on every quant format (full-precision
32/32, int4 11/32, int2 1/32, mix 5/32 -- byte-identical to the pre-change
binary); fmt=4 grouped path preserved (the scale check is by construction the
same condition detect_group_size already imposed); a hand-crafted hostile
safetensors is refused cleanly; ASan+UBSan clean on legit and hostile loads
(only the pre-existing intentional startup leaks remain).
These are the C trust-boundary items of #368, landed as a minimal standalone
fix; the server-side and build items of that PR follow via its rebase.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
colibri/_version.py now reads c/version.py (#394's single source of truth --
coli --version, the release workflow, and pip metadata can no longer drift),
with an importlib.metadata fallback for the installed-wheel case where c/ is
not on disk. README documents that pip install -e . is the supported form:
the engine lives in c/ and is not packaged into a standalone wheel.
Verified in a clean venv: pip install -e . -> colibri.__version__ == 1.0.0
read from c/version.py, coli entrypoint on PATH and functional.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
GitHub Actions shells are format strings; the bare 'msys2' string made the
v1.0.0 tag build fail before its first step ('Invalid shell option'). Same
invocation ci.yml already uses. path-type: inherit so the Package step can
reach the runner's 7z.exe.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
cap_for_ram's projection is an estimate: on the GB10 (#403) long generations
overshot it by ~40 GB (projected 74.4, real 115.6) and the kernel killed the
engine three times. Run D of the issue proves a low cap CONTAINS the growth;
this guard does that automatically, keyed on MEASURED RSS instead of the
projection.
At the repin safe point (no moe in flight), every ~16 emitted tokens: if RSS
exceeds the resolved budget (RAM_GB/auto, or an explicit RSS_GUARD_GB
ceiling), free the least-used LRU expert slabs in place and lower ecap so the
cache cannot regrow. Slabs are >128 KB so glibc returns the pages to the
kernel immediately -- RSS actually drops.
Eviction never compacts the array: with PILOT_REAL the pilot worker holds
pointers into ecache[] across its preads, so the slot stays in place with
eid=-1/used=0 (first candidate for reuse); reserved slots (eid<0) are never
touched and victim selection happens under g_pilot_mx. resident_bytes is left
alone: LRU slots are never accounted there (only pin + dense).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Vincenzo Fornaro