pin_load computed npin from the RAM budget FIRST, then carved the
VRAM-ranked prefix out of those npin slots. With CUDA_RELEASE_HOST the
prefix's host slabs are freed right after upload — so on a multi-GPU
host the top-ranked experts consumed the RAM budget without occupying
RAM, and the CPU tier pinned only the leftovers. Measured on 6x RTX 5090
(251 GB): 9,280 VRAM + only 1,721 RAM pins (32.5 GB warm) on a box whose
RAM fits ~10k more — the cold tail then paid disk on every token and the
hit rate ceilinged at 99.0% forever.
Move the VRAM budget estimate above the npin finalization and make the
release-destined prefix ADDITIVE to the RAM-derived count. Same box,
same env plus the fix:
[PIN] placement: 9,280 VRAM + 10,176 RAM (192.5 GB warm)
expert hit rate 100.0% (pin 100.0% + lru 0.0%) — disk 0
1,024-token greedy decode: 3.62 -> 6.21 tok/s (+72%)
warm late segment (t=768-1024): 5.11 -> 5.73 tok/s (+12%)
prefill: 10.4 -> 8.8 s
The whole-run gain is the LRU warmup penalty disappearing (full
residency from the first token); the late-segment gain is the steady
~0.9%-miss disk residue recovered. Non-release configs are untouched:
prefix_est stays 0 and the arithmetic reduces to today's exactly.
Re-derivation of e7/disk-class-instr @ de6dd6d (base caa49f7) onto
origin/dev @ 61004dc, after PR #391 split c/glm.c into c/colibri.c plus
quant.h/sample.h/kv_persist.h/telemetry.h/grammar.h. Semantic equivalence,
not a copy: same events, same accounting, re-sited onto the new tree.
Placement: colibri.c, unchanged from before the split. telemetry.h (#391)
is the dashboard/stats module (HWINFO/TIERS/EMAP/HITS protocol lines +
usage persistence) — prof_report(), expert_load_impl(), moe(), g_prof_io
and g_edisk_ns all stayed in colibri.c, so DISK-CLASS's aggregation and
printing follow them there. No relocation needed, no DEVIATION.
Read path: #362 (prefetcher-v3, 22509fc) turned out to be testbed-only
scope per its own merge message ("glm.c untouched") — confirmed zero
c/glm.c changes in that merge's diff. The seven expert_load() call sites
this patch touches (pipe_worker, expert_host_ensure, moe()'s OMP miss
loop, pilot_realload, repin_pass_limit, pin_load x2) are structurally
identical to the pre-split tree; the demand flag re-attaches at the same
sites with the same semantics (1 only at moe()'s own PIPE/OMP miss path,
0 everywhere else), so DISK-CLASS still counts demand loads only.
One real drift, unrelated to #362: #417 (cfcc742) fixed the exact "Metal
pre-routed FASE A never bumps the real elast/eaccess_clock" defect this
feature's comments described as a documented, deliberately-unfixed
upstream issue — the real clock now ticks in FASE A too. The private
elast_dc/eaccess_clock_dc clock is kept anyway: its job was never only
to route around that freeze, it also snapshots pre-bump state so a
call's own routing bump can't contaminate its own classification, and
keeping DISK-CLASS's bookkeeping fully separate from stock elast state
is what makes "byte-identical with PROF=0" provable by construction
instead of by argument. Code comments referencing the old defect are
updated to reflect the fix (historical note + #417/cfcc742 pointer)
rather than describing a bug that no longer exists.
Gates: make glm METAL=0 and METAL=1 both clean, zero warnings (matches
stock 61004dc, also built clean with zero warnings for comparison).
make test-c: 0 failures. make test-python: 77 tests, OK.
Authored by Fable 5 in Claude Code, analysis in partnership with
@Monotophic.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The has_mtp completeness probe checked for `mlp.experts.255.down_proj.weight`,
which only exists when n_routed_experts == 256. REAP-pruned checkpoints (and any
MoE with a different expert count) have fewer experts, so the probe spuriously
reported has_mtp=0 and disabled MTP speculative decode even though the head was
present and complete. Probe `mlp.experts.<n_experts-1>` instead.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rename glm.c → colibri.c and extract four self-contained modules
into header-only files (same pattern as st.h/tier.h/grammar.h):
quant.h (672 lines) — SIMD matmul kernels, quantization
sample.h (143 lines) — RNG, top-p sampling, stop-set
kv_persist.h (121 lines) — .coli_kv disk persistence
telemetry.h (189 lines) — dashboard protocol, stats, usage
Main engine file shrinks from 6588 to 5396 lines (−18%).
Build system: primary target is now colibri$(EXE); `make glm`
remains as a phony alias for backward compat. CI, setup.sh,
coli CLI, and all 10 test files that include the engine are
updated. make check passes (C + Python, 73 tests, zero warnings).
Vincenzo Fornaro