1. DEFENSIVE (backend_metal.mm, coli_metal_register): re-registering a live
base overwrote s.buf and resset_add'ed the new wrapper without removing
the OLD one from the residency set -- ARC drops our reference but the set
retains the object and keeps its pages resident: a leak and a
set/g_slabs divergence. The found branch now stashes the replaced
wrapper under g_slab_mtx and, after dropping the lock (round-1 invariant:
no Metal call under the slab lock), resset_remove(old)s it before
resset_add(b); identical old==b early-outs both set operations.
Invariant defended, stated in the comment: residency-set membership
mirrors g_slabs exactly. No in-tree caller re-registers a live base
today (audited) -- closed defensively.
2. DOC (SUMMARY.md): the moe_submit lifecycle bullet still said
resset_flush commits "under g_slab_mtx" -- stale text from before the
round-1 mutex split; the code takes g_resset_mtx and never touches the
slab lock there. Parenthetical corrected; register bullet updated to
describe the re-register hygiene from item 1.
expert_load's fslab-OOM unwind freed s->slab via compat_aligned_free without
coli_metal_unregister -- a pre-existing gap on main/dev (validator-confirmed
during E4) that leaves a stale g_slabs entry over freed memory, letting
resolve() hand the GPU a dangling pointer. Under COLI_METAL_RESSET=1 the
exposure is strictly longer-lived: the wrapped buffer would stay a permanent
residency-set member over the freed pages instead of stock's transient
last-command-buffer window. Ported from e4/metal-heap validator fix 6753225,
adapted to dev's non-heap code shape (no coli_metal_heap_free wrapper --
plain unregister-before-free).
The uring_load_add analog (E4 audit round-2 insurance) is deliberately not
carried: that arm is #ifdef __linux__-gated while COLI_METAL is macOS-only,
so it is dead code on every real build target, and unlike E4 this branch has
no allocation-path reason to touch the function.
Reachable only through allocation failure mid-load; verified by inspection
and clean builds (no OOM-injection harness in tree). SUMMARY.md item 10
updated to "carried on this branch".
1. REQUIRED (backend_metal.mm): no Metal call runs under g_slab_mtx anymore.
The set mutations (addAllocation/removeAllocation/commit) and the dirty
flag moved to a dedicated g_resset_mtx; coli_metal_register/unregister do
their g_slabs bookkeeping under g_slab_mtx exactly as stock, then call
resset_add/resset_remove after dropping it (still before returning, so
unregister's remove+commit still lands before the caller frees the host
memory). The original shape -- commit under the slab lock the parallel OMP
loader threads contend on -- was structurally identical to the mutex-over-
live-Metal-call bug E4's audit round 2 fixed, the leading suspect for its
replicated +12s expert-disk regression. The register->flush->resolve
happens-before argument survives the split: resset_add completes inside
coli_metal_register before it returns, the engine cannot dispatch an
expert before its load returns, and add/flush are serialized by
g_resset_mtx (comment at resset_add). The two mutexes are never held
together, so no lock-order hazard exists.
2. REQUIRED (backend_metal.mm, backend_metal.h, glm.c): the resset_flush
cost in moe_submit -- deliberately outside the g_t_setup window so the
A/B harness counters keep their meaning -- was invisible. New
g_t_resset_flush accumulator timed around the flush, exported via
coli_metal_resset_stats(), printed by profile_print as a separate
gate-on-only "METAL-RESSET: flush" line (mirrors E4's METAL-HEAP line;
the METAL: line the harness parses keeps its exact format; stock output
byte-identical -- the stats call returns 0 with the gate off and the
whole block sits in the pre-existing #ifdef COLI_METAL arm). Register-
side add/remove costs land in the existing t_ewait window; comment at
resset_add says so instead of a second counter.
3. REQUIRED (SUMMARY.md; the moe_submit commit message was already reworded
in place, pre-push): corrected two false attributions -- the hazard-
tracking and thread-safety statements were credited to the SDK header,
which is silent on both; the actual source is Apple's online
MTLResidencySet class reference and residency-set adoption guide
(fetched 2026-07-18). Note: the scaffolding commit's message was checked
and contains no such claim, so it was left untouched.
4. DOC (SUMMARY.md): the pre-existing fslab OOM-unwind bug (glm.c ~1868-73,
frees s->slab without unregister) has a strictly longer-lived exposure
under E5 (permanent set member vs stock's transient per-CB declaration).
Out of scope here; the upstream PR built from E5 must carry the one-line
fix (reference: E4 branch commit 6753225).
The one seam the mechanism history actually implicates: moe_submit's `use`
vector (resolved expert weight/scale slabs) is the only useResource: loop
whose length scales with LRU cache size. With COLI_METAL_RESSET=1, skip that
loop entirely -- the queue-attached MTLResidencySet already guarantees those
buffers are resident -- after resset_flush() commits any pending adds from a
loader burst. Every other useResource: call site (bind_gemv's weight/scale
buffers, attn_decode/layer_decode's Lb/Rb/kvbW/kvbS/inB/pnB/rwB/rbB,
coli_metal_gemm's wb/sb) is left unconditionally unchanged: none of them
scale with cache size, and Lb/Rb carry real GPU-side write traffic ordered by
existing explicit memoryBarrierWithScope: calls, not by useResource:'s hazard
tracking -- narrowing the blast radius rather than removing useResource:
uniformly. Full reasoning, the hazard-tracking tradeoff (residency sets don't
support hazard tracking, per Apple's MTLResidencySet developer documentation
and residency-set adoption guide; the SDK header itself is silent on the
topic), and every judgment call in UNCERTAINTIES: see SUMMARY.md.
Gate off is unaffected: g_resset_enabled is false, so the useResource: loop
runs exactly as before.
Env-gated (COLI_METAL_RESSET=1) persistent MTLResidencySet attached to g_queue
(macOS 15+, @available-guarded with a one-line stderr fallback). Adds
resset_add/resset_remove/resset_flush helpers, wires them into the existing
coli_metal_register/coli_metal_unregister/coli_metal_shutdown bodies -- no new
functions in backend_metal.h, no glm.c changes, every existing call site keeps
its current signature and behavior.
register() defers the commit (resset_add just marks dirty, under the same
g_slab_mtx that already serializes parallel OMP loader threads) so a loader
burst doesn't pay a commit per slab; unregister() commits synchronously and
immediately, because the caller frees the underlying host memory right after
it returns and a deferred removal would leave the set referencing freed
memory. Nothing yet reads g_resset_enabled to change dispatch behavior --
this commit is bookkeeping only, gate on or off, so it does not change what
any command buffer does (verified by inspection: no useResource:/useHeap:
call site is touched here).
Gate off (default) is byte-for-byte the stock path: g_resset_enabled starts
false and nothing sets it outside the COLI_METAL_RESSET branch in
coli_metal_init, so every new helper is a no-op.
See SUMMARY.md (next commit) for the full design and UNCERTAINTIES.
Lightweight i18n without react-i18next: a LocaleProvider context +
useLocale() hook with {{var}} interpolation, auto-detect from
navigator.language, persisted to localStorage.
98 translation keys across 4 locale files (en/zh-CN/zh-TW/it).
All user-visible strings in App/Brain/Profiling/ErrorBoundary are
now t() calls. Language switcher added to the sidebar footer.
Build clean (tsc + vite), 18 tests pass.
New files:
README.zh-CN.md — simplified Chinese (大陆用词)
README.it.md — Italian (the project's "mother tongue")
All four READMEs now link to each other in a consistent nav bar.
Updated zh-TW to reflect glm.c → colibri.c rename and new headers.
Rename glm.c → colibri.c and extract four self-contained modules
into header-only files (same pattern as st.h/tier.h/grammar.h):
quant.h (672 lines) — SIMD matmul kernels, quantization
sample.h (143 lines) — RNG, top-p sampling, stop-set
kv_persist.h (121 lines) — .coli_kv disk persistence
telemetry.h (189 lines) — dashboard protocol, stats, usage
Main engine file shrinks from 6588 to 5396 lines (−18%).
Build system: primary target is now colibri$(EXE); `make glm`
remains as a phony alias for backward compat. CI, setup.sh,
coli CLI, and all 10 test files that include the engine are
updated. make check passes (C + Python, 73 tests, zero warnings).
On Metal, when routing is precomputed on the GPU (g_pre_idx), the moe fast
path bumps eusage/ehit/eheat for the selected experts but skips the one thing
the full CPU router does at the equivalent site: elast[layer][e] =
++eaccess_clock. So the session-local recency clock advances during prefill
(full router) but freezes the moment GPU-prerouted decode starts, and REPIN's
tier_pick_lfru() tie-breaker then runs on stale recency for the rest of the
run. Mirror the exact update the non-Metal path already does. Inside
#ifdef COLI_METAL, so CPU/CUDA are untouched; elast only feeds the LFRU
eviction heuristic, so this cannot affect output, only which experts REPIN
keeps warm.
Found and reported by @monotophic with a source-level trace repro
(ELAST_TRACE). Fix is inspection-verified against line ~3055; needs a
Metal build to exercise end-to-end.
Closes#417
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds docs/quickstart.md — a step-by-step, no-experience-assumed walkthrough
from installing the build tools to the first coli chat, with per-OS
copy-paste commands (Ubuntu apt, Windows MSYS2 or prebuilt binary, macOS
brew), the ready-made HF int4 container plus the self-convert path, and an
honest 'what to expect' on disk-bound speed. Commands verified against
setup.sh and the coli subcommands; every cross-linked doc exists. Linked
from the README's Get started section.
Closes#414
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Colibri loads model directories and safetensors from mirrors it does not
control, so the file's declared shapes and byte spans are attacker-influenced
input. Three memory-safety holes on that boundary, independently confirmed
(incl. a from-scratch adversarial audit that re-derived the same two) and
present in the shipped v1.0.0:
- st.h st_read_f32: numel came from the shape, nbytes from the offsets, with
no cross-check. A crafted tensor whose shape inflates numel past nbytes made
the BF16/F16 loop read past the malloc'd raw buffer and the F32 memcpy write
past the caller's config-sized destination (heap OOB read + write). Now
enforce numel*esz == nbytes before any copy.
- st.h header parse: the shape product could overflow int64 to a small/negative
numel that would then pass the cross-check. Guard each multiply.
- glm.c qt_resolve_fmt (new, replaces the three duplicated "?1:?2:3" fmt sites
in qt_from_disk and both expert_load arms): the old inference SILENTLY fell
to int2 for any unrecognized weight byte count, so a too-short weight became
a valid int2 whose matmul read O*I nibbles past the buffer; and an oversized
scale array overflowed the per-row t->s. Now the weight bytes must match a
known int8/int4/int2 layout and the scale array must match the expected
per-row (O) or grouped (O*ng) cardinality, else refuse.
- glm.c config/generation_config slurp: unbounded ftell -> malloc(n+1) gave a
hostile file a load-time OOM, and on malloc failure b[got]=0 was a NULL
deref. Cap at 256 MB and NULL-check.
Verified: TF token-exactness unchanged on every quant format (full-precision
32/32, int4 11/32, int2 1/32, mix 5/32 -- byte-identical to the pre-change
binary); fmt=4 grouped path preserved (the scale check is by construction the
same condition detect_group_size already imposed); a hand-crafted hostile
safetensors is refused cleanly; ASan+UBSan clean on legit and hostile loads
(only the pre-existing intentional startup leaks remain).
These are the C trust-boundary items of #368, landed as a minimal standalone
fix; the server-side and build items of that PR follow via its rebase.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
colibri/_version.py now reads c/version.py (#394's single source of truth --
coli --version, the release workflow, and pip metadata can no longer drift),
with an importlib.metadata fallback for the installed-wheel case where c/ is
not on disk. README documents that pip install -e . is the supported form:
the engine lives in c/ and is not packaged into a standalone wheel.
Verified in a clean venv: pip install -e . -> colibri.__version__ == 1.0.0
read from c/version.py, coli entrypoint on PATH and functional.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
GitHub Actions shells are format strings; the bare 'msys2' string made the
v1.0.0 tag build fail before its first step ('Invalid shell option'). Same
invocation ci.yml already uses. path-type: inherit so the Package step can
reach the runner's 7z.exe.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
cap_for_ram's projection is an estimate: on the GB10 (#403) long generations
overshot it by ~40 GB (projected 74.4, real 115.6) and the kernel killed the
engine three times. Run D of the issue proves a low cap CONTAINS the growth;
this guard does that automatically, keyed on MEASURED RSS instead of the
projection.
At the repin safe point (no moe in flight), every ~16 emitted tokens: if RSS
exceeds the resolved budget (RAM_GB/auto, or an explicit RSS_GUARD_GB
ceiling), free the least-used LRU expert slabs in place and lower ecap so the
cache cannot regrow. Slabs are >128 KB so glibc returns the pages to the
kernel immediately -- RSS actually drops.
Eviction never compacts the array: with PILOT_REAL the pilot worker holds
pointers into ecache[] across its preads, so the slot stays in place with
eid=-1/used=0 (first candidate for reuse); reserved slots (eid<0) are never
touched and victim selection happens under g_pilot_mx. resident_bytes is left
alone: LRU slots are never accounted there (only pin + dense).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CreateProcess cannot exec a shebang script, so the gateway exits during
setUpClass on the windows CI job. The gateway logic under test is
platform-independent and stays covered by the POSIX jobs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The gateway's tool-calling path had unit coverage (parse_tool_calls,
render_chat) but nothing exercised the real subprocess wire protocol or the
HTTP surface a coding client actually hits. #401 reports plain-text replies
where tool_calls were expected; every documented path checks out, so pin the
whole path down with a mock engine speaking SUBMIT/DATA/DONE and assert:
- non-stream: tool_calls populated, finish_reason tool_calls, no raw markers
- stream: markers suppressed across 20-way chunk splits, tool_calls delta
- tool-result round trip: <|observation|><tool_response> rendering, text reply
- no tools: plain text untouched
Also emit a stderr diagnosis when tools are declared and tool-call markers
are present in the reply but the strict parse matches nothing (typically
quantization-mangled output) pointing at COLI_TOOL_SALVAGE=1 -- the likely
field condition behind #401.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fix(test): make test_stops build on Windows (mkdtemp compat shim)
Conflict with #366's getenv_utf8 in compat.h resolved by keeping both blocks.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both coli's cuda_binary() and doctor.py's cuda_linkage() detect CUDA support
by running `ldd` on the engine binary and looking for a linked libcudart —
but that's Linux-only (cuda_binary() checks sys.platform != "linux", and
cuda_linkage() checks os.name != "posix", both short-circuiting to False on
win32). Windows CUDA_DLL=1 builds never link libcudart at all: glm.exe loads
coli_cuda.dll dynamically via LoadLibrary at startup (backend_loader.c), so
there's no import-table entry for ldd/dumpbin to find in the first place.
The practical effect: `coli doctor` always reported "NVIDIA GPU detected but
the engine is CPU-only" on Windows, and `coli run/chat/serve --gpu ...`
always hard-exited with "--gpu needs the CUDA build" — even on a correctly
built CUDA_DLL=1 binary with coli_cuda.dll sitting right next to glm.exe.
Fix: on win32, detect a COLI_CUDA build by scanning glm.exe for the marker
string "[CUDA] mode: routed experts", which only exists in code compiled
under #ifdef COLI_CUDA (see glm.c's cuda init block), then confirm
coli_cuda.dll actually sits next to the binary — mirroring the Linux
"linked but missing" distinction. Linux/macOS detection is unchanged.
Verified on Windows 11 with mingw-w64 GCC 16.1.0 + CUDA 13.2 + RTX 5080:
`coli doctor` now reports "CUDA engine and devices are available", and
`coli run --gpu 0 --vram 8` populates VRAM (confirmed via the engine's own
"[CUDA] resident set: N tensors, X GB VRAM" runtime log) instead of exiting.
- c/version.py: single source of truth (__version__ = "1.0.0")
- coli: reads version.py, banner shows dynamic version, --version flag
- .github/workflows/release.yml: tag push triggers cross-platform build
(Linux x86_64, macOS ARM64, Windows x86_64) and creates a GitHub
Release with packaged binaries + changelog notes
- CHANGELOG.md: v1.0.0 baseline documenting all shipped features
To cut a release:
1. bump c/version.py
2. add a CHANGELOG section
3. git tag v1.0.0 && git push --tags
Per-row int4 quantization of the MTP head's eh_proj [D,2D] zeroes its ENTIRE
embedding half: the tensor's two column halves differ in scale by ~20-30x per
row (embedding-half absmax ~0.05, hidden-half ~1.5 on GLM-5.2), so a single
per-row scale (absmax/7) puts every embedding-half weight below half a
quantization step and np.rint rounds it to exact zero (packed bytes 0x88).
The draft head then cannot see the input token and MTP acceptance collapses
to ~0% — the mechanism behind issue #8's measurement (int4: 0-4%; int8:
39-59%), and the reason --mtp already defaults to --ebits 8.
This showed up in the wild: a published pre-converted container had its MTP
shards made with an explicit --ebits 4 and drafts were pure garbage on every
backend (deterministic, data-side; verified byte-for-byte by reproducing the
published bytes with quant_int4 on the official BF16 rows).
- tools/repair_mtp_int8.py: repairs such a container IN PLACE — finds the MTP
layer's per-row-int4 dense tensors (eh_proj, q/kv/o projections, shared
experts; routed experts untouched), re-downloads only those from the FP8
source repo (~355 MB of HTTP range reads, no torch, no token), requantizes
at int8 with quant_int8's exact math, and rewrites the shards atomically
with *.bak-int4 backups. --dry-run to inspect. The engine picks up int8
automatically (qt_from_disk detects format by blob size).
Validated on GLM-5.2 744B: MTP acceptance 0% -> 100% (greedy), 1.11 -> 3.20
tokens/forward, decode 0.18 -> 0.30 tok/s (Metal, 4-bit KV).
- convert_fp8_to_int4.py: print a warning when --mtp is combined with
--ebits <8 and per-row scales (the exact footgun above); --group-size 128
remains a valid int4 alternative since group scales give the embedding
half its own scale.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two gaps in the local path, both raised in #383:
- The main --indir pass copied only config.json while the download path
copies four files; without tokenizer.json the converted container can't
run chat/serve. Copy the same four, print what was copied, and warn
when the source is missing any (tokenizer.json called out explicitly).
- Local passes restarted from shard 0 on every rerun. out-NNNNN names
count EMITTED shards, not input indexes (shards with no relevant
tensors emit nothing), so the download path's exists-check can't be
mirrored directly: a sidecar manifest per prefix records
input -> output (or empty) plus the conversion parameters, written
atomically after each shard. Rerun skips what matches and refuses a
parameter mismatch on the same outdir instead of mixing containers
(the #355 failure mode).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
coli run hardcoded the nothink template (<|assistant|><think></think>),
so THINK=1 had no effect in one-shot runs while serve mode honors it
(glm.c serve template picks <think> vs <think></think> from the same
env var). Pick the template the same way here.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
When --indir contains model.safetensors.index.json, the head/indexer
passes convert only the shards that actually hold the requested tensors
(3 instead of scanning 141 for --mtp — every empty scan still opens a
5 GB shard). Prints the selection in the [PLAN] block. Without the index
the full scan is unchanged.
Output is byte-identical to the unfiltered path (sha256-verified on the
GLM-5.2 FP8 head shards, with a decoy shard proving the filter excludes
rather than reorders).
Co-Authored-By: Claude <noreply@anthropic.com>
Two operator-surprise fixes:
openai_server.py (#382, LordMZTE): a request without max_tokens defaulted to
min(256, limit) — so `coli serve --ngen 32768` still cut every answer at 256
tokens for clients that omit the field. The operator's configured budget IS the
default now; generation still ends at EOS, so it's a cap, not a target.
convert_fp8_to_int4.py (#383, bokiko): the resolved conversion plan (mode,
source, ebits/io/x bits, grouped-vs-per-row) prints as a [PLAN] line before any
work. The two traps in #383 — --mtp defaulting ebits to 8 with the grouped
branch silently gated off, and --mtp appearing to ignore --indir — are already
defused by the #355 fix (the --mtp pass emits ONLY head tensors on the local
path), but a 3.5-hour job must show its plan at second 1, not in a post-hoc
size sanity check. bokiko's exact invocation now prints:
[PLAN] mode: MTP head only | source: local ./fp8 | experts 8-bit, ... |
PER-ROW (grouped branch needs bits<=4; ebits=8 disables it)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
yurivict connected crush per the walkthrough and got 'thought for 1h8m and did
nothing' — which is not a hang: agent CLIs send a 10-20k-token system preamble,
and prefill on the CPU-streaming path runs at a few tok/s (attention-bound,
#153). An hour of silent prefill looks exactly like a dead server. The note now
spells out the arithmetic, the curl smoke-test that separates slow-but-working
from broken, and honest guidance on what agent workloads are (not) viable.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
monotophic