yurivict connected crush per the walkthrough and got 'thought for 1h8m and did
nothing' — which is not a hang: agent CLIs send a 10-20k-token system preamble,
and prefill on the CPU-streaming path runs at a few tok/s (attention-bound,
#153). An hour of silent prefill looks exactly like a dead server. The note now
spells out the arithmetic, the curl smoke-test that separates slow-but-working
from broken, and honest guidance on what agent workloads are (not) viable.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
docs/WINDOWS.md (the #329 install walkthrough) and docs/windows.md (the concise
reference from the README restructure) differ only in case. On case-insensitive
filesystems — every default Windows and macOS checkout — the two paths collide:
the working tree can never be clean, and `git rebase`/`git status` refuse to
run. @KingIcyCreamProjects reported it.
Everything links to the lowercase docs/windows.md (README.md, README.zh-TW.md,
docs/cuda.md), so that is the canonical name. This keeps it and folds in both
contents: the step-by-step walkthrough (download-first, Smart App Control, CUDA
DLL, first run, reference numbers, failure index) followed by the build-flags
reference (AVX-VNNI, warmup). docs/WINDOWS.md is removed. No links change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Documents connecting OpenAI-compatible coding CLIs to `coli serve`.
Covers the common snag reported in #373 — clients like crush refuse to
start without an API key even though the local endpoint needs none — by
showing that any dummy key works (Colibri only enforces COLI_API_KEY if
set). Concrete recipes for aider and crush, a curl smoke test, the
generic base-URL/model/key pattern for other tools, and an honest
tok/s-latency caveat for the streaming path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ENVIRONMENT.md says it is "generated from source by scanning every
getenv() site", but it had drifted far behind the code. Reconciling the
doc against the C sources (the MAINTAINING-DOCS.md procedure) found 34
engine env vars read by the code but undocumented, and nothing stale.
Added (defaults/effects taken from source, per the maintenance rules -
nothing invented):
Performance/tuning (11): COLI_NUMA, PILOT_TWO, COUPLE/COUPLE_K/COUPLE_D,
ROUTE_TRACE, COLI_NO_FUSED_PAIR, DISK_SPLIT, I4S, SPEC_PIN,
COLI_RAM_OVERCOMMIT
CUDA (16): COLI_CUDA_ATTN_SHARD, COLI_CUDA_PIPE/_PIPE_SHARD/_PIPE_S_MIN,
COLI_CUDA_MTP, COLI_CUDA_ASYNC, COLI_CUDA_DUAL_PROJ, COLI_CUDA_W4_PACKED,
COLI_CUDA_TC_INT4/_TC_MIN_ROWS/_TC_W4A16/_TC_W4A16_MIN,
COLI_CUDA_SHARED_W4A16/_SHARED_W4A16_MIN_ROWS, COLI_METAL_UNTRACKED
Advanced/debug (7): SCHEMA, EXPERT_BUDGET/_EXPERIMENTAL, TOKENS,
SCORE_PREFIX, REPIN_VERBOSE, PPL (olmoe-only), COLI_PROMPT (CLI section)
Also bumped the "Generated from" line to dev @ d5327e2 and noted the scan
now covers olmoe.c, backend_cuda.cu, backend_metal.mm (not just glm.c).
Verified: the code-vs-doc diff is now empty - all 111 distinct C env vars
are documented. The reverse diff (doc vars not in C code) is 14 entries,
all legitimate: 11 Python-side vars correctly in the Server/CLI section,
plus 3 prose constants (IOSQE_ASYNC, O_DIRECT, the VAR format word).
Answers 'where does it slow down on THIS machine with THIS config' so users
can tune RAM_GB/PIPE/DIRECT/PIN for their hardware without folklore:
- startup header: CPU/cores/RAM/backend + effective knobs (cache cap, pin,
DRAFT/PIPE/DIRECT/MMAP/IDOT/DSA/PILOT/CACHE_ROUTE) — every saved log is
self-describing when comparing runs across configs or machines
- per-forward decode latency ring (32k) -> p50/p90/p99/max, plus a tail
diagnosis when p99 >> p50 (cold-cache expert loads)
- expert I/O accounting at the pread/mmap-touch sites: GB fetched, MB/token,
GB/s, hit rate, loads/token, pinned/LRU tier fill
- phase shares of wall time and a plain-language verdict naming the knob
most likely to move tok/s (I/O-bound vs compute-bound vs attention-bound)
- reports after REPLAY / PROMPT / oracle runs (stdout) and per turn in serve
mode (stderr; stdout stays the framed protocol)
Additive only: with PROF unset every mode's output is byte-identical.
hwinfo_emit's /proc probe is factored into hw_probe() and shared.
Validated end-to-end on a tiny-random unquantized fixture (REPLAY, PROF on/
off, RAM_GB squeeze flips hit 96.9%->26.6% and the verdict follows);
make check clean, 0 warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TCBNCciBaHea41QmidLUMn
PIN=auto resolves to <model>/.coli_usage, the history that serve mode
appends after every turn, so each restart's hot-store placement follows the
accumulated REAL workload instead of a frozen one-shot profile; stats.txt is
the fallback for a virgin model dir, and with neither present the run simply
starts unpinned. Same magic-value convention as PIN_GB=all (#80); explicit
paths and the AUTOPIN flow are untouched (PIN set skips AUTOPIN as before).
This lifts deployment entrypoints' "prefer .coli_usage over stats.txt" shell
plumbing into the engine, plus the ENVIRONMENT.md row.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Paper-style cache-aware MoE selection (arXiv:2412.00099 max-rank):
keep true top-J always; fill remaining K slots preferring experts already
resident in pin∪LRU within top-M. Default OFF so stock full top-K is
unchanged.
Env: CACHE_ROUTE, ROUTE_J/M/P/ALPHA, ROUTE_AGREE (auto-on with CACHE_ROUTE).
Telemetry: swap%/route_swaps/route_slots, route_agree, route_kl on footer
and serve STAT. Complementary to PILOT (prefetch vs selection change).
Routing-only PR for clean A/B vs PILOT / #119; no CUDA/fuse stack.
See docs/CACHE_ROUTE.md. Closes nothing; for #161 discussion.
Co-authored-by: Vincent Marquez <vincentmarquez405@gmail.com>
- 'See it running' right after the intro: the live dashboard (744B
answering at 4+ tok/s end-to-end with metrics and tier bar) and the
Brain cortex (19,456 experts, tier colour, routing heat, per-turn
firing) — three seconds to understand what colibri is.
- A short table of contents for the long read below.
- 'Web dashboard' section: coli web one-liner, what each panel shows,
link to the Expert Atlas (#175). Existing content untouched.
Canonical write-up of the GRAMMAR= draft source: mechanism, why it pays in a
disk-streaming MoE specifically, usage/knobs, measured expectations by workload
shape (span-density dependence, from the #146 A/Bs incl. corrections), lossless
guarantees, bench discipline, prior art. Linked from the README feature bullet.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Docs-only. Documents that the OMP active-spin steals SoC power from the Metal GPU on Apple Silicon (default regresses -39%); COLI_NO_OMP_TUNE=1 + PIPE=1 recovers and beats the pre-rebase branch (2.24 vs 2.06 tok/s). Flags a follow-up: Metal builds should default to passive OMP wait.
Records the July 12-13 lab findings on one 6x RTX 5090 machine:
- vLLM-Moet comparison and why per-rank expert residency dominates
- colibri full-resident placement: 2.30 -> 6.28-6.84 tok/s, with the
tested-and-rejected directions documented
- MTP speculation: broken int4 head identified (issue #8), int8 head
reaches 69-79% acceptance but MoE verify batches scale expert cost
with S, so speculation loses at every depth; revisit after GPU
grouped GEMM
- AVX-512 int4 kernel qualification: numerically better than the old
order, quality-neutral (ppl delta 0.24%), +4-7% on CPU-heavy routing
README links the record from the honest-numbers section.
ZacharyZcR