Answers 'where does it slow down on THIS machine with THIS config' so users
can tune RAM_GB/PIPE/DIRECT/PIN for their hardware without folklore:
- startup header: CPU/cores/RAM/backend + effective knobs (cache cap, pin,
DRAFT/PIPE/DIRECT/MMAP/IDOT/DSA/PILOT/CACHE_ROUTE) — every saved log is
self-describing when comparing runs across configs or machines
- per-forward decode latency ring (32k) -> p50/p90/p99/max, plus a tail
diagnosis when p99 >> p50 (cold-cache expert loads)
- expert I/O accounting at the pread/mmap-touch sites: GB fetched, MB/token,
GB/s, hit rate, loads/token, pinned/LRU tier fill
- phase shares of wall time and a plain-language verdict naming the knob
most likely to move tok/s (I/O-bound vs compute-bound vs attention-bound)
- reports after REPLAY / PROMPT / oracle runs (stdout) and per turn in serve
mode (stderr; stdout stays the framed protocol)
Additive only: with PROF unset every mode's output is byte-identical.
hwinfo_emit's /proc probe is factored into hw_probe() and shared.
Validated end-to-end on a tiny-random unquantized fixture (REPLAY, PROF on/
off, RAM_GB squeeze flips hit 96.9%->26.6% and the verdict follows);
make check clean, 0 warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TCBNCciBaHea41QmidLUMn
PIN=auto resolves to <model>/.coli_usage, the history that serve mode
appends after every turn, so each restart's hot-store placement follows the
accumulated REAL workload instead of a frozen one-shot profile; stats.txt is
the fallback for a virgin model dir, and with neither present the run simply
starts unpinned. Same magic-value convention as PIN_GB=all (#80); explicit
paths and the AUTOPIN flow are untouched (PIN set skips AUTOPIN as before).
This lifts deployment entrypoints' "prefer .coli_usage over stats.txt" shell
plumbing into the engine, plus the ENVIRONMENT.md row.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Paper-style cache-aware MoE selection (arXiv:2412.00099 max-rank):
keep true top-J always; fill remaining K slots preferring experts already
resident in pin∪LRU within top-M. Default OFF so stock full top-K is
unchanged.
Env: CACHE_ROUTE, ROUTE_J/M/P/ALPHA, ROUTE_AGREE (auto-on with CACHE_ROUTE).
Telemetry: swap%/route_swaps/route_slots, route_agree, route_kl on footer
and serve STAT. Complementary to PILOT (prefetch vs selection change).
Routing-only PR for clean A/B vs PILOT / #119; no CUDA/fuse stack.
See docs/CACHE_ROUTE.md. Closes nothing; for #161 discussion.
Co-authored-by: Vincent Marquez <vincentmarquez405@gmail.com>
- 'See it running' right after the intro: the live dashboard (744B
answering at 4+ tok/s end-to-end with metrics and tier bar) and the
Brain cortex (19,456 experts, tier colour, routing heat, per-turn
firing) — three seconds to understand what colibri is.
- A short table of contents for the long read below.
- 'Web dashboard' section: coli web one-liner, what each panel shows,
link to the Expert Atlas (#175). Existing content untouched.
Canonical write-up of the GRAMMAR= draft source: mechanism, why it pays in a
disk-streaming MoE specifically, usage/knobs, measured expectations by workload
shape (span-density dependence, from the #146 A/Bs incl. corrections), lossless
guarantees, bench discipline, prior art. Linked from the README feature bullet.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Docs-only. Documents that the OMP active-spin steals SoC power from the Metal GPU on Apple Silicon (default regresses -39%); COLI_NO_OMP_TUNE=1 + PIPE=1 recovers and beats the pre-rebase branch (2.24 vs 2.06 tok/s). Flags a follow-up: Metal builds should default to passive OMP wait.
Records the July 12-13 lab findings on one 6x RTX 5090 machine:
- vLLM-Moet comparison and why per-rank expert residency dominates
- colibri full-resident placement: 2.30 -> 6.28-6.84 tok/s, with the
tested-and-rejected directions documented
- MTP speculation: broken int4 head identified (issue #8), int8 head
reaches 69-79% acceptance but MoE verify batches scale expert cost
with S, so speculation loses at every depth; revisit after GPU
grouped GEMM
- AVX-512 int4 kernel qualification: numerically better than the old
order, quality-neutral (ppl delta 0.24%), +4-7% on CPU-heavy routing
README links the record from the honest-numbers section.
JustVugg