Upstream courtesy fix, found while auditing the conversion recipe for this
branch -- pre-existing in dev, not introduced by int3-g64, but directly
relevant to anyone converting for real (a #383-class gap: same failure
family as the resume/manifest work already done for --indir, and the
silent-mixing mechanism issue #355 fixed for a narrower case).
The --indir path already refuses to resume with different conversion
parameters on the same --outdir (a manifest records ebits/xbits/io_bits/
group_size/n_layers/bits_map and compares on every resume). The --repo
streaming download loops (main model, --mtp, --indexer) never got the
same guard: each shard's resume check is just `if os.path.exists(outp):
continue` -- true whether or not THIS run's flags match the flags that
produced that shard. A --repo conversion resumed with changed bits
(--xbits 3 -> 4 mid-run, say, after an interruption) would silently mix
bit-widths across shards in the same container, with no error and no log
line distinguishing it from a normal resume.
Fix: check_or_record_params(), a small shared helper mirroring the
--indir manifest's refuse-on-mismatch logic but without needing its
per-shard bookkeeping (the --repo loops already track shard completion
correctly via out-NNNNN.safetensors existence, since shard index maps
directly to output filename there -- only whether the params used SO FAR
still match needed adding). Applied to all three --repo loops with
per-mode sidecar files (.out-mtp-params.json / .out-idx-params.json /
.out-params.json) so a --mtp and a main-model conversion into the same
--outdir don't cross-check each other's parameters.
Also includes PROJ_BITS (the per-projection expert bit overrides) in the
tracked params dict on BOTH paths -- it was missing from --indir's
existing manifest too, so a resume with a changed --up-bits/--gate-bits/
--down-bits would have passed the existing guard silently.
Verified directly (no real HF downloads; --repo network paths can't be
exercised under this task's constraints): unit-tested
check_or_record_params() standalone -- fresh outdir accepts and records,
a same-params resume accepts, a changed --xbits is refused, and a
proj_bits-only change (nothing else different) is refused. Re-ran the
existing --indir dry-run end to end (convert, resume, resume-with-changed-
xbits) to confirm the manifest-based path still works correctly with
proj_bits added to its params dict.
Gates: make test-c (20/20) and make test-python (85/85) both pass.
rss_guard marked the victim slot eid=-1 under the lock, unlocked, and only then
freed s->slab. In that window the slot reads as {eid=-1, slab still valid} --
exactly the state pilot_realload's victim scan reuses first -- so a pilot worker
could claim it and pread into the slab while it was being freed: use-after-free,
or a double-free if the loader took its own realloc path.
Keep the free and the pointer/capacity NULLing inside the critical section, so
'slab valid' and 'slot reusable' are never simultaneously observable. The lock is
held across a free() (microseconds); workers only take it briefly for scan+reserve.
Reproduces on dev with a SINGLE pilot worker (rss_guard runs on the main thread
while the pilot worker runs in the background), whenever the RAM guard is active
(RSS_GUARD_GB, or any resolved g_ram_budget_gb).
make check 83/83, native + portable builds 0 warnings.
pin_load computed npin from the RAM budget FIRST, then carved the
VRAM-ranked prefix out of those npin slots. With CUDA_RELEASE_HOST the
prefix's host slabs are freed right after upload — so on a multi-GPU
host the top-ranked experts consumed the RAM budget without occupying
RAM, and the CPU tier pinned only the leftovers. Measured on 6x RTX 5090
(251 GB): 9,280 VRAM + only 1,721 RAM pins (32.5 GB warm) on a box whose
RAM fits ~10k more — the cold tail then paid disk on every token and the
hit rate ceilinged at 99.0% forever.
Move the VRAM budget estimate above the npin finalization and make the
release-destined prefix ADDITIVE to the RAM-derived count. Same box,
same env plus the fix:
[PIN] placement: 9,280 VRAM + 10,176 RAM (192.5 GB warm)
expert hit rate 100.0% (pin 100.0% + lru 0.0%) — disk 0
1,024-token greedy decode: 3.62 -> 6.21 tok/s (+72%)
warm late segment (t=768-1024): 5.11 -> 5.73 tok/s (+12%)
prefill: 10.4 -> 8.8 s
The whole-run gain is the LRU warmup penalty disappearing (full
residency from the first token); the late-segment gain is the steady
~0.9%-miss disk residue recovered. Non-release configs are untouched:
prefix_est stays 0 and the arithmetic reduces to today's exactly.
Re-derivation of e7/disk-class-instr @ de6dd6d (base caa49f7) onto
origin/dev @ 61004dc, after PR #391 split c/glm.c into c/colibri.c plus
quant.h/sample.h/kv_persist.h/telemetry.h/grammar.h. Semantic equivalence,
not a copy: same events, same accounting, re-sited onto the new tree.
Placement: colibri.c, unchanged from before the split. telemetry.h (#391)
is the dashboard/stats module (HWINFO/TIERS/EMAP/HITS protocol lines +
usage persistence) — prof_report(), expert_load_impl(), moe(), g_prof_io
and g_edisk_ns all stayed in colibri.c, so DISK-CLASS's aggregation and
printing follow them there. No relocation needed, no DEVIATION.
Read path: #362 (prefetcher-v3, 22509fc) turned out to be testbed-only
scope per its own merge message ("glm.c untouched") — confirmed zero
c/glm.c changes in that merge's diff. The seven expert_load() call sites
this patch touches (pipe_worker, expert_host_ensure, moe()'s OMP miss
loop, pilot_realload, repin_pass_limit, pin_load x2) are structurally
identical to the pre-split tree; the demand flag re-attaches at the same
sites with the same semantics (1 only at moe()'s own PIPE/OMP miss path,
0 everywhere else), so DISK-CLASS still counts demand loads only.
One real drift, unrelated to #362: #417 (cfcc742) fixed the exact "Metal
pre-routed FASE A never bumps the real elast/eaccess_clock" defect this
feature's comments described as a documented, deliberately-unfixed
upstream issue — the real clock now ticks in FASE A too. The private
elast_dc/eaccess_clock_dc clock is kept anyway: its job was never only
to route around that freeze, it also snapshots pre-bump state so a
call's own routing bump can't contaminate its own classification, and
keeping DISK-CLASS's bookkeeping fully separate from stock elast state
is what makes "byte-identical with PROF=0" provable by construction
instead of by argument. Code comments referencing the old defect are
updated to reflect the fix (historical note + #417/cfcc742 pointer)
rather than describing a bug that no longer exists.
Gates: make glm METAL=0 and METAL=1 both clean, zero warnings (matches
stock 61004dc, also built clean with zero warnings for comparison).
make test-c: 0 failures. make test-python: 77 tests, OK.
Authored by Fable 5 in Claude Code, analysis in partnership with
@Monotophic.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
GLM-5.2's config defines three stop tokens: <|endoftext|>, <|user|>, and
<|observation|>. In serve mode, when the model generates <tool_call> blocks,
int4-quantized logit noise can cause argmax to pick a <|user|> or
<|observation|> token ID, immediately stopping generation.
The <|user|> and <|observation|> tokens are role markers handled by the
Python API server, not the C engine. Filter them out in stops_arm() when
SERVE=1, keeping only <|endoftext|> as the stop token.
- Add SERVE-mode guard in stops_arm() that retains only the EOS token
- Log the number of filtered tokens for diagnostics
The generation() method was re-extracting tools from the raw request body,
bypassing the filtering that render_chat() applies when tool_choice is a
function object (forced function call). This caused parse_tool_calls() to
receive the unfiltered tool list, producing incorrect or missing tool_calls
in the response.
- Add tools and tool_choice parameters to generation() signature
- Pass them from chat_completion() after render_chat() processing
- Add early structural validation of tools in generation_options()
for clear HTTP 400 errors on malformed input
The has_mtp completeness probe checked for `mlp.experts.255.down_proj.weight`,
which only exists when n_routed_experts == 256. REAP-pruned checkpoints (and any
MoE with a different expert count) have fewer experts, so the probe spuriously
reported has_mtp=0 and disabled MTP speculative decode even though the head was
present and complete. Probe `mlp.experts.<n_experts-1>` instead.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extend resource_plan to classify the hardware into bottleneck regimes
(disk / memory / mixed / compute) and derive tuning knobs automatically:
MTP: off when compute-bound (42% loss at full residency, #389)
or disk-bound with <90% hit (union growth adds reads)
PIPE: COLI_CUDA_PIPE=1 single-GPU, =2 multi-GPU, PIPE=1 CPU disk
NUMA: selective interleave for GPU hosts, blanket hint for CPU-only
PIN: PIN_GB=all when fully resident + no GPU
OMP: COLI_NO_OMP_TUNE=1 for Metal (spin steals GPU power)
`coli plan` now shows an auto-tune section with each knob and its
reason. `environment_for_plan()` applies them via setdefault so
explicit user settings always win.
plan version stays at 2 (additive fields: bottleneck_class,
projected_hit_rate, tune). 7 new tests covering all regimes.
Lightweight i18n without react-i18next: a LocaleProvider context +
useLocale() hook with {{var}} interpolation, auto-detect from
navigator.language, persisted to localStorage.
98 translation keys across 4 locale files (en/zh-CN/zh-TW/it).
All user-visible strings in App/Brain/Profiling/ErrorBoundary are
now t() calls. Language switcher added to the sidebar footer.
Build clean (tsc + vite), 18 tests pass.
New files:
README.zh-CN.md — simplified Chinese (大陆用词)
README.it.md — Italian (the project's "mother tongue")
All four READMEs now link to each other in a consistent nav bar.
Updated zh-TW to reflect glm.c → colibri.c rename and new headers.
Rename glm.c → colibri.c and extract four self-contained modules
into header-only files (same pattern as st.h/tier.h/grammar.h):
quant.h (672 lines) — SIMD matmul kernels, quantization
sample.h (143 lines) — RNG, top-p sampling, stop-set
kv_persist.h (121 lines) — .coli_kv disk persistence
telemetry.h (189 lines) — dashboard protocol, stats, usage
Main engine file shrinks from 6588 to 5396 lines (−18%).
Build system: primary target is now colibri$(EXE); `make glm`
remains as a phony alias for backward compat. CI, setup.sh,
coli CLI, and all 10 test files that include the engine are
updated. make check passes (C + Python, 73 tests, zero warnings).
On Metal, when routing is precomputed on the GPU (g_pre_idx), the moe fast
path bumps eusage/ehit/eheat for the selected experts but skips the one thing
the full CPU router does at the equivalent site: elast[layer][e] =
++eaccess_clock. So the session-local recency clock advances during prefill
(full router) but freezes the moment GPU-prerouted decode starts, and REPIN's
tier_pick_lfru() tie-breaker then runs on stale recency for the rest of the
run. Mirror the exact update the non-Metal path already does. Inside
#ifdef COLI_METAL, so CPU/CUDA are untouched; elast only feeds the LFRU
eviction heuristic, so this cannot affect output, only which experts REPIN
keeps warm.
Found and reported by @monotophic with a source-level trace repro
(ELAST_TRACE). Fix is inspection-verified against line ~3055; needs a
Metal build to exercise end-to-end.
Closes#417
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds docs/quickstart.md — a step-by-step, no-experience-assumed walkthrough
from installing the build tools to the first coli chat, with per-OS
copy-paste commands (Ubuntu apt, Windows MSYS2 or prebuilt binary, macOS
brew), the ready-made HF int4 container plus the self-convert path, and an
honest 'what to expect' on disk-bound speed. Commands verified against
setup.sh and the coli subcommands; every cross-linked doc exists. Linked
from the README's Get started section.
Closes#414
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Colibri loads model directories and safetensors from mirrors it does not
control, so the file's declared shapes and byte spans are attacker-influenced
input. Three memory-safety holes on that boundary, independently confirmed
(incl. a from-scratch adversarial audit that re-derived the same two) and
present in the shipped v1.0.0:
- st.h st_read_f32: numel came from the shape, nbytes from the offsets, with
no cross-check. A crafted tensor whose shape inflates numel past nbytes made
the BF16/F16 loop read past the malloc'd raw buffer and the F32 memcpy write
past the caller's config-sized destination (heap OOB read + write). Now
enforce numel*esz == nbytes before any copy.
- st.h header parse: the shape product could overflow int64 to a small/negative
numel that would then pass the cross-check. Guard each multiply.
- glm.c qt_resolve_fmt (new, replaces the three duplicated "?1:?2:3" fmt sites
in qt_from_disk and both expert_load arms): the old inference SILENTLY fell
to int2 for any unrecognized weight byte count, so a too-short weight became
a valid int2 whose matmul read O*I nibbles past the buffer; and an oversized
scale array overflowed the per-row t->s. Now the weight bytes must match a
known int8/int4/int2 layout and the scale array must match the expected
per-row (O) or grouped (O*ng) cardinality, else refuse.
- glm.c config/generation_config slurp: unbounded ftell -> malloc(n+1) gave a
hostile file a load-time OOM, and on malloc failure b[got]=0 was a NULL
deref. Cap at 256 MB and NULL-check.
Verified: TF token-exactness unchanged on every quant format (full-precision
32/32, int4 11/32, int2 1/32, mix 5/32 -- byte-identical to the pre-change
binary); fmt=4 grouped path preserved (the scale check is by construction the
same condition detect_group_size already imposed); a hand-crafted hostile
safetensors is refused cleanly; ASan+UBSan clean on legit and hostile loads
(only the pre-existing intentional startup leaks remain).
These are the C trust-boundary items of #368, landed as a minimal standalone
fix; the server-side and build items of that PR follow via its rebase.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
colibri/_version.py now reads c/version.py (#394's single source of truth --
coli --version, the release workflow, and pip metadata can no longer drift),
with an importlib.metadata fallback for the installed-wheel case where c/ is
not on disk. README documents that pip install -e . is the supported form:
the engine lives in c/ and is not packaged into a standalone wheel.
Verified in a clean venv: pip install -e . -> colibri.__version__ == 1.0.0
read from c/version.py, coli entrypoint on PATH and functional.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
GitHub Actions shells are format strings; the bare 'msys2' string made the
v1.0.0 tag build fail before its first step ('Invalid shell option'). Same
invocation ci.yml already uses. path-type: inherit so the Package step can
reach the runner's 7z.exe.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
cap_for_ram's projection is an estimate: on the GB10 (#403) long generations
overshot it by ~40 GB (projected 74.4, real 115.6) and the kernel killed the
engine three times. Run D of the issue proves a low cap CONTAINS the growth;
this guard does that automatically, keyed on MEASURED RSS instead of the
projection.
At the repin safe point (no moe in flight), every ~16 emitted tokens: if RSS
exceeds the resolved budget (RAM_GB/auto, or an explicit RSS_GUARD_GB
ceiling), free the least-used LRU expert slabs in place and lower ecap so the
cache cannot regrow. Slabs are >128 KB so glibc returns the pages to the
kernel immediately -- RSS actually drops.
Eviction never compacts the array: with PILOT_REAL the pilot worker holds
pointers into ecache[] across its preads, so the slot stays in place with
eid=-1/used=0 (first candidate for reuse); reserved slots (eid<0) are never
touched and victim selection happens under g_pilot_mx. resident_bytes is left
alone: LRU slots are never accounted there (only pin + dense).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CreateProcess cannot exec a shebang script, so the gateway exits during
setUpClass on the windows CI job. The gateway logic under test is
platform-independent and stays covered by the POSIX jobs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The gateway's tool-calling path had unit coverage (parse_tool_calls,
render_chat) but nothing exercised the real subprocess wire protocol or the
HTTP surface a coding client actually hits. #401 reports plain-text replies
where tool_calls were expected; every documented path checks out, so pin the
whole path down with a mock engine speaking SUBMIT/DATA/DONE and assert:
- non-stream: tool_calls populated, finish_reason tool_calls, no raw markers
- stream: markers suppressed across 20-way chunk splits, tool_calls delta
- tool-result round trip: <|observation|><tool_response> rendering, text reply
- no tools: plain text untouched
Also emit a stderr diagnosis when tools are declared and tool-call markers
are present in the reply but the strict parse matches nothing (typically
quantization-mangled output) pointing at COLI_TOOL_SALVAGE=1 -- the likely
field condition behind #401.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fix(test): make test_stops build on Windows (mkdtemp compat shim)
Conflict with #366's getenv_utf8 in compat.h resolved by keeping both blocks.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both coli's cuda_binary() and doctor.py's cuda_linkage() detect CUDA support
by running `ldd` on the engine binary and looking for a linked libcudart —
but that's Linux-only (cuda_binary() checks sys.platform != "linux", and
cuda_linkage() checks os.name != "posix", both short-circuiting to False on
win32). Windows CUDA_DLL=1 builds never link libcudart at all: glm.exe loads
coli_cuda.dll dynamically via LoadLibrary at startup (backend_loader.c), so
there's no import-table entry for ldd/dumpbin to find in the first place.
The practical effect: `coli doctor` always reported "NVIDIA GPU detected but
the engine is CPU-only" on Windows, and `coli run/chat/serve --gpu ...`
always hard-exited with "--gpu needs the CUDA build" — even on a correctly
built CUDA_DLL=1 binary with coli_cuda.dll sitting right next to glm.exe.
Fix: on win32, detect a COLI_CUDA build by scanning glm.exe for the marker
string "[CUDA] mode: routed experts", which only exists in code compiled
under #ifdef COLI_CUDA (see glm.c's cuda init block), then confirm
coli_cuda.dll actually sits next to the binary — mirroring the Linux
"linked but missing" distinction. Linux/macOS detection is unchanged.
Verified on Windows 11 with mingw-w64 GCC 16.1.0 + CUDA 13.2 + RTX 5080:
`coli doctor` now reports "CUDA engine and devices are available", and
`coli run --gpu 0 --vram 8` populates VRAM (confirmed via the engine's own
"[CUDA] resident set: N tensors, X GB VRAM" runtime log) instead of exiting.
- c/version.py: single source of truth (__version__ = "1.0.0")
- coli: reads version.py, banner shows dynamic version, --version flag
- .github/workflows/release.yml: tag push triggers cross-platform build
(Linux x86_64, macOS ARM64, Windows x86_64) and creates a GitHub
Release with packaged binaries + changelog notes
- CHANGELOG.md: v1.0.0 baseline documenting all shipped features
To cut a release:
1. bump c/version.py
2. add a CHANGELOG section
3. git tag v1.0.0 && git push --tags
Vincenzo Fornaro