The has_mtp completeness probe checked for `mlp.experts.255.down_proj.weight`,
which only exists when n_routed_experts == 256. REAP-pruned checkpoints (and any
MoE with a different expert count) have fewer experts, so the probe spuriously
reported has_mtp=0 and disabled MTP speculative decode even though the head was
present and complete. Probe `mlp.experts.<n_experts-1>` instead.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A second (read-only) copy of the model on another drive is registered as a
per-shard read replica; expert loads are split between the drives by a
deterministic (layer,eid) hash weighted by COLI_DISK_WEIGHTS=<primary>,<mirror>
or, by default, by a startup bandwidth probe using the engine's own access
pattern (parallel ~19 MB reads, O_DIRECT). Cold decode is disk-bound
(~11 GB/token), so two NVMe queues add up.
- st.h: mirror accepted per file only if size + safetensors header are
byte-identical to the primary (identical data_offsets by construction, so
every pread is valid on either copy); partial mirrors work (smaller second
SSD holding only some shards); the mirror is never written — .coli_usage /
.coli_kv stay on the primary.
- glm.c: routing covers the coalesced slab pread, O_DIRECT, mmap/Metal and
scale reads, plus the OMP-parallel pin/autopin warmup (streams from both
drives). Deterministic routing keeps readahead/PILOT WILLNEED on the same
drive as the demand read and avoids caching an expert twice. A mirror read
error falls back to the primary (one warning, no crash). Per-drive bytes
are reported in a MIRROR: stats line.
- tests: test_st_mirror covers validation, read equality on both replicas,
and the rejection paths (divergent header, size mismatch, missing file).
Measured on 2x NVMe (GLM-5.2 int4, greedy, DRAFT=0, DIRECT=1, cold-ish
cache): 0.42 -> 0.57 tok/s (+36%), expert-disk service 15.2s -> 10.3s,
byte-identical output; probe chose a 48/52 split.
Extend resource_plan to classify the hardware into bottleneck regimes
(disk / memory / mixed / compute) and derive tuning knobs automatically:
MTP: off when compute-bound (42% loss at full residency, #389)
or disk-bound with <90% hit (union growth adds reads)
PIPE: COLI_CUDA_PIPE=1 single-GPU, =2 multi-GPU, PIPE=1 CPU disk
NUMA: selective interleave for GPU hosts, blanket hint for CPU-only
PIN: PIN_GB=all when fully resident + no GPU
OMP: COLI_NO_OMP_TUNE=1 for Metal (spin steals GPU power)
`coli plan` now shows an auto-tune section with each knob and its
reason. `environment_for_plan()` applies them via setdefault so
explicit user settings always win.
plan version stays at 2 (additive fields: bottleneck_class,
projected_hit_rate, tune). 7 new tests covering all regimes.
Lightweight i18n without react-i18next: a LocaleProvider context +
useLocale() hook with {{var}} interpolation, auto-detect from
navigator.language, persisted to localStorage.
98 translation keys across 4 locale files (en/zh-CN/zh-TW/it).
All user-visible strings in App/Brain/Profiling/ErrorBoundary are
now t() calls. Language switcher added to the sidebar footer.
Build clean (tsc + vite), 18 tests pass.
New files:
README.zh-CN.md — simplified Chinese (大陆用词)
README.it.md — Italian (the project's "mother tongue")
All four READMEs now link to each other in a consistent nav bar.
Updated zh-TW to reflect glm.c → colibri.c rename and new headers.
Rename glm.c → colibri.c and extract four self-contained modules
into header-only files (same pattern as st.h/tier.h/grammar.h):
quant.h (672 lines) — SIMD matmul kernels, quantization
sample.h (143 lines) — RNG, top-p sampling, stop-set
kv_persist.h (121 lines) — .coli_kv disk persistence
telemetry.h (189 lines) — dashboard protocol, stats, usage
Main engine file shrinks from 6588 to 5396 lines (−18%).
Build system: primary target is now colibri$(EXE); `make glm`
remains as a phony alias for backward compat. CI, setup.sh,
coli CLI, and all 10 test files that include the engine are
updated. make check passes (C + Python, 73 tests, zero warnings).
On Metal, when routing is precomputed on the GPU (g_pre_idx), the moe fast
path bumps eusage/ehit/eheat for the selected experts but skips the one thing
the full CPU router does at the equivalent site: elast[layer][e] =
++eaccess_clock. So the session-local recency clock advances during prefill
(full router) but freezes the moment GPU-prerouted decode starts, and REPIN's
tier_pick_lfru() tie-breaker then runs on stale recency for the rest of the
run. Mirror the exact update the non-Metal path already does. Inside
#ifdef COLI_METAL, so CPU/CUDA are untouched; elast only feeds the LFRU
eviction heuristic, so this cannot affect output, only which experts REPIN
keeps warm.
Found and reported by @monotophic with a source-level trace repro
(ELAST_TRACE). Fix is inspection-verified against line ~3055; needs a
Metal build to exercise end-to-end.
Closes#417
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds docs/quickstart.md — a step-by-step, no-experience-assumed walkthrough
from installing the build tools to the first coli chat, with per-OS
copy-paste commands (Ubuntu apt, Windows MSYS2 or prebuilt binary, macOS
brew), the ready-made HF int4 container plus the self-convert path, and an
honest 'what to expect' on disk-bound speed. Commands verified against
setup.sh and the coli subcommands; every cross-linked doc exists. Linked
from the README's Get started section.
Closes#414
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This sets the formatter to alejandra. I was trying to stay as close as
the original as I could; otherwise we could set it to `nixfmt` which
would keep list spacing more similar, but would introduce additional
indentation in a few places.
Colibri loads model directories and safetensors from mirrors it does not
control, so the file's declared shapes and byte spans are attacker-influenced
input. Three memory-safety holes on that boundary, independently confirmed
(incl. a from-scratch adversarial audit that re-derived the same two) and
present in the shipped v1.0.0:
- st.h st_read_f32: numel came from the shape, nbytes from the offsets, with
no cross-check. A crafted tensor whose shape inflates numel past nbytes made
the BF16/F16 loop read past the malloc'd raw buffer and the F32 memcpy write
past the caller's config-sized destination (heap OOB read + write). Now
enforce numel*esz == nbytes before any copy.
- st.h header parse: the shape product could overflow int64 to a small/negative
numel that would then pass the cross-check. Guard each multiply.
- glm.c qt_resolve_fmt (new, replaces the three duplicated "?1:?2:3" fmt sites
in qt_from_disk and both expert_load arms): the old inference SILENTLY fell
to int2 for any unrecognized weight byte count, so a too-short weight became
a valid int2 whose matmul read O*I nibbles past the buffer; and an oversized
scale array overflowed the per-row t->s. Now the weight bytes must match a
known int8/int4/int2 layout and the scale array must match the expected
per-row (O) or grouped (O*ng) cardinality, else refuse.
- glm.c config/generation_config slurp: unbounded ftell -> malloc(n+1) gave a
hostile file a load-time OOM, and on malloc failure b[got]=0 was a NULL
deref. Cap at 256 MB and NULL-check.
Verified: TF token-exactness unchanged on every quant format (full-precision
32/32, int4 11/32, int2 1/32, mix 5/32 -- byte-identical to the pre-change
binary); fmt=4 grouped path preserved (the scale check is by construction the
same condition detect_group_size already imposed); a hand-crafted hostile
safetensors is refused cleanly; ASan+UBSan clean on legit and hostile loads
(only the pre-existing intentional startup leaks remain).
These are the C trust-boundary items of #368, landed as a minimal standalone
fix; the server-side and build items of that PR follow via its rebase.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
colibri/_version.py now reads c/version.py (#394's single source of truth --
coli --version, the release workflow, and pip metadata can no longer drift),
with an importlib.metadata fallback for the installed-wheel case where c/ is
not on disk. README documents that pip install -e . is the supported form:
the engine lives in c/ and is not packaged into a standalone wheel.
Verified in a clean venv: pip install -e . -> colibri.__version__ == 1.0.0
read from c/version.py, coli entrypoint on PATH and functional.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
GitHub Actions shells are format strings; the bare 'msys2' string made the
v1.0.0 tag build fail before its first step ('Invalid shell option'). Same
invocation ci.yml already uses. path-type: inherit so the Package step can
reach the runner's 7z.exe.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
cap_for_ram's projection is an estimate: on the GB10 (#403) long generations
overshot it by ~40 GB (projected 74.4, real 115.6) and the kernel killed the
engine three times. Run D of the issue proves a low cap CONTAINS the growth;
this guard does that automatically, keyed on MEASURED RSS instead of the
projection.
At the repin safe point (no moe in flight), every ~16 emitted tokens: if RSS
exceeds the resolved budget (RAM_GB/auto, or an explicit RSS_GUARD_GB
ceiling), free the least-used LRU expert slabs in place and lower ecap so the
cache cannot regrow. Slabs are >128 KB so glibc returns the pages to the
kernel immediately -- RSS actually drops.
Eviction never compacts the array: with PILOT_REAL the pilot worker holds
pointers into ecache[] across its preads, so the slot stays in place with
eid=-1/used=0 (first candidate for reuse); reserved slots (eid<0) are never
touched and victim selection happens under g_pilot_mx. resident_bytes is left
alone: LRU slots are never accounted there (only pin + dense).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CreateProcess cannot exec a shebang script, so the gateway exits during
setUpClass on the windows CI job. The gateway logic under test is
platform-independent and stays covered by the POSIX jobs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The gateway's tool-calling path had unit coverage (parse_tool_calls,
render_chat) but nothing exercised the real subprocess wire protocol or the
HTTP surface a coding client actually hits. #401 reports plain-text replies
where tool_calls were expected; every documented path checks out, so pin the
whole path down with a mock engine speaking SUBMIT/DATA/DONE and assert:
- non-stream: tool_calls populated, finish_reason tool_calls, no raw markers
- stream: markers suppressed across 20-way chunk splits, tool_calls delta
- tool-result round trip: <|observation|><tool_response> rendering, text reply
- no tools: plain text untouched
Also emit a stderr diagnosis when tools are declared and tool-call markers
are present in the reply but the strict parse matches nothing (typically
quantization-mangled output) pointing at COLI_TOOL_SALVAGE=1 -- the likely
field condition behind #401.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fix(test): make test_stops build on Windows (mkdtemp compat shim)
Conflict with #366's getenv_utf8 in compat.h resolved by keeping both blocks.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ZacharyZcR