Both coli's cuda_binary() and doctor.py's cuda_linkage() detect CUDA support
by running `ldd` on the engine binary and looking for a linked libcudart —
but that's Linux-only (cuda_binary() checks sys.platform != "linux", and
cuda_linkage() checks os.name != "posix", both short-circuiting to False on
win32). Windows CUDA_DLL=1 builds never link libcudart at all: glm.exe loads
coli_cuda.dll dynamically via LoadLibrary at startup (backend_loader.c), so
there's no import-table entry for ldd/dumpbin to find in the first place.
The practical effect: `coli doctor` always reported "NVIDIA GPU detected but
the engine is CPU-only" on Windows, and `coli run/chat/serve --gpu ...`
always hard-exited with "--gpu needs the CUDA build" — even on a correctly
built CUDA_DLL=1 binary with coli_cuda.dll sitting right next to glm.exe.
Fix: on win32, detect a COLI_CUDA build by scanning glm.exe for the marker
string "[CUDA] mode: routed experts", which only exists in code compiled
under #ifdef COLI_CUDA (see glm.c's cuda init block), then confirm
coli_cuda.dll actually sits next to the binary — mirroring the Linux
"linked but missing" distinction. Linux/macOS detection is unchanged.
Verified on Windows 11 with mingw-w64 GCC 16.1.0 + CUDA 13.2 + RTX 5080:
`coli doctor` now reports "CUDA engine and devices are available", and
`coli run --gpu 0 --vram 8` populates VRAM (confirmed via the engine's own
"[CUDA] resident set: N tensors, X GB VRAM" runtime log) instead of exiting.
- c/version.py: single source of truth (__version__ = "1.0.0")
- coli: reads version.py, banner shows dynamic version, --version flag
- .github/workflows/release.yml: tag push triggers cross-platform build
(Linux x86_64, macOS ARM64, Windows x86_64) and creates a GitHub
Release with packaged binaries + changelog notes
- CHANGELOG.md: v1.0.0 baseline documenting all shipped features
To cut a release:
1. bump c/version.py
2. add a CHANGELOG section
3. git tag v1.0.0 && git push --tags
coli run hardcoded the nothink template (<|assistant|><think></think>),
so THINK=1 had no effect in one-shot runs while serve mode honors it
(glm.c serve template picks <think> vs <think></think> from the same
env var). Pick the template the same way here.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
No more manual pkill: cmd_serve writes a pidfile, cmd_stop finds the server
(pidfile, then /proc cmdline) and its engine (comm glm/exe/olmoe with SERVE=1
in environ — the engine re-execs for OMP tuning so its comm is 'exe', which is
why every 'pkill -x glm' in history killed nothing). SIGTERM, wait, SIGKILL.
--dry-run lists targets without acting. First real use took down a live
serve+engine pair cleanly and released 16 GB.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit 41a872c331a2a0a8655699e0171c68dd2bcda186)
The cold-chat cost, measured on this box: every `coli chat` spawns a private
engine (34-136 s of resident load) and starts with an empty expert cache (hit
4% cold vs 55% warm). Quitting throws both away.
`coli chat` now probes localhost:8000 first (~1 ms when nothing listens) and,
if a `coli serve` answers, runs the REPL over plain OpenAI SSE against it:
stdlib urllib only, engine byte-protocol untouched. --attach [URL] forces it,
--no-attach restores a private engine. reasoning_content keepalive pings are
filtered; :reset starts a new conversation client-side (the server's KV slots
reuse prefixes per conversation on their own).
Verified against a mock SSE server (pings ignored, markdown rendered, :reset,
clean exit) and against the real 744B model: two consecutive sessions, second
attach instant with zero reload — the engine stayed resident at 15.7 GB across
both. Honest limit: warmth carry-over BETWEEN different conversations is small
here because cap=3 slots/layer is a short memory; the structural wins are the
load never being repaid and same-conversation continuation.
LOCAL ONLY for now — not pushed, per the current working rule.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit aa406ccab8a4501b924bc3a9f4725dd1d18a685d)
Two silent failures that compound into "[engine terminated]" with no cause.
cap_for_ram() floors the expert cache at cap=1. When resident+slack have
already blown the budget, avail is negative and capmax would be 0 -- "I do
not fit in your budget". Flooring it to 1 and carrying on turns "I do not
fit" into "I overshoot", which is precisely the mid-generation OOM-kill the
function exists to prevent: it printed "projected peak 25.1 GB" against a
22 GB budget and started anyway. Now it says so, names PIN_GB when that is
what inflated the resident set, and refuses to start when the peak also
exceeds the memory actually available on the machine (COLI_RAM_OVERCOMMIT=1
overrides).
The kernel kills with SIGKILL: no error, no log, stdout just closes. coli
read that EOF and printed "[engine terminated]" without ever reaping the
child, so an OOM-kill was indistinguishable from a clean exit -- the report
in #305 (Debian 12, dies mid-generation, no message). engine_diag() now
reports the signal or exit code, names the OOM-killer when it was SIGKILL,
and shows the tail of the engine's stderr.
Reported-by: Ne00n <#305>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
One HF 504 killed the whole bench. Now: load_dataset retries with
exponential backoff (hf_hub resumes partial downloads from cache); a task
that still fails is skipped instead of killing the rest; JSONLs are written
atomically (coli only checks existence, so a truncated file from an
interrupted run would block re-download forever); coli bench drops
still-missing tasks with a warning and refuses to run eval with none.
Co-Authored-By: Claude <noreply@anthropic.com>
On Windows the engine self-exec OMP tuning never runs (Linux/FreeBSD-only)
and posix_fadvise readahead is a compat.h no-op, so a stock Windows run
leaves large measured wins on the table. The launcher now setdefaults, on
win32 only, each independently overridable by setting the variable:
- OMP_WAIT_POLICY=active, GOMP_SPINCOUNT=200000, OMP_DYNAMIC=FALSE,
OMP_NUM_THREADS=<physical cores> (parity with the glm.c self-exec block;
COLI_NO_OMP_TUNE disables exactly this block, presence-based like the
engine). OMP_PROC_BIND/OMP_PLACES deliberately omitted and also removed
from environment_for_plan on win32: MinGW libgomp has no affinity support
("Affinity not supported on this configuration").
- DIRECT=1: unbuffered expert reads. Measured on a 9950X3D + Samsung 9100
PRO Gen5 + Win11: iobench 10.68 GB/s O_DIRECT vs 9.03 buffered (warm);
end-to-end REPLAY 0.48 -> 1.02 tok/s. Matches #162 (1.47x same class).
- PIPE=1: load/matmul overlap, byte-identical output; +8% on top of DIRECT
(PIPE_WORKERS untouched at 8 - 4/8/16 swept flat on Gen5).
- PILOT_REAL=1: real cross-layer prefetch, the only working prefetch on
Windows; +11% and expert hit rate +19 points.
Full ladder methodology and numbers: 96-token greedy REPLAY, one lever per
step, medians of 3-4 runs (see the fork tuning doc referenced in the PR).
tests/test_env_defaults.py covers the defaults, explicit-override-wins,
the kill-switch scope, and the non-win32 no-op.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
After the READY handshake, run_serve emits one TIERS status line (the
web-dashboard expert-pyramid snapshot). cmd_chat never read it, so in
interactive chat it surfaced as literal "TIERS 0 181 19275 0.00 ..." text
prepended to the first response. Read and discard that one line after READY,
matching how the other status frames are consumed. Chat-only; the HTTP serve
path already drains it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
In serve/chat mode, Ctrl-C during generation killed the whole engine, losing
the loaded model and forcing a full reload. Now a SIGINT handler (armed only
in run_serve / run_serve_mux) sets a flag that spec_decode's token loop treats
exactly like hitting the NGEN cap: the turn ends through the normal path, so
the END sentinel, STAT line, usage_save and KV append all run — and :more can
continue the interrupted answer. The mux loop closes in-flight requests via the
same mux_done path. One-shot ./glm runs and Windows keep default SIGINT (die).
coli: stream_turn survives the first KeyboardInterrupt, forwards SIGINT to the
engine (covers non-TTY), drains to the turn boundary, and reports it. A second
Ctrl-C quits. Help line and per-turn footer updated.
POSIX only (sigaction); no behaviour change on Windows. Verified end-to-end on
Apple M4 + Metal: interrupt mid-decode, engine stays up, next prompt answers,
:q exits 0.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
cmd_bench builds the eval_glm.py command but did not pass --glm,
so eval_glm.py fell back to ./glm which breaks when running from
any directory other than the source tree. Pass the already-resolved
GLM variable via --glm, matching --data and --snap.
Assisted-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Benchmark datasets are downloaded artifacts, not source files. Store them
under $XDG_CACHE_HOME/colibri/bench (~/.cache/colibri/bench) on Linux/macOS
and %LOCALAPPDATA%\colibri\bench on Windows instead of polluting the source
tree at c/bench/.
fetch_benchmarks.py already calls os.makedirs(out, exist_ok=True), so the
new cache path is created on first use.
Assisted-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Web dashboard: live expert-tier panel (VRAM/RAM/disk), static hosting, coli web command
- Engine: TIERS protocol line (counts + GB for VRAM/RAM/disk experts)
emitted at READY and refreshed after every served turn, computed from
the live pin/LRU/CUDA state.
- Server: dispatcher tracks the latest snapshot, /health exposes it as
"tiers"; the built web UI (web/dist) is served from the same port
(read-only, traversal-safe, SPA fallback) so the dashboard needs no
second process.
- Web: tier bar + legend in the Runtime panel showing where the 19k
experts live right now, with GB per tier; updates with the existing
health polling.
- CLI: `coli web` = serve + auto-open the browser once /health answers
(the 744B load takes minutes; a poller waits for it), --no-browser to
disable, friendly hint when web/dist isn't built.
* coli web: HERE is a path string, not a Path
* web: same-origin default + auto-connect when served by the engine
The page hosted by coli web talked to http://127.0.0.1:8000/v1 by
default while being loaded from http://localhost:8000 - a different
origin, so the very first probe died on CORS ('Failed to fetch').
When the page is engine-served the default (and a stored stale factory
default) now resolve to window.location.origin + /v1, and the console
auto-probes on load; the Vite dev server keeps the classic default.
The server's own port is also whitelisted in DEFAULT_CORS_ORIGINS for
the localhost/127.0.0.1 cross-name case.
* web: fix auto-connect effect returning Promise (React 19 StrictMode crash)
The async connect() call inside useEffect returned a Promise that React
19 StrictMode stored as the effect cleanup. On re-render, React called
destroy_() on that Promise — 'not a function'. Block body with explicit
return undefined prevents any non-function from leaking into the cleanup
slot.
* web: rich streaming metrics — live token counter, tok/s gauge, TTFT, session totals
During generation: a flashing Zap badge counts tokens in real time.
After: Gauge shows tok/s, Timer shows TTFT (time to first token),
Layers shows prompt→completion token counts, Clock shows queue wait.
Session totals (cumulative prompt + completion) appear in the Runtime
panel. All with lucide icons and tabular-nums for stable layout.
Tier bar gains a smoother cubic-bezier transition and slightly taller
height for visual weight.
* web: hardware environment panel — CPU model, GPU count/VRAM, RAM total/free, cores
Engine emits HWINFO protocol line at startup (CPU name from /proc/cpuinfo,
core count, RAM total/available from /proc/meminfo, GPU count + VRAM from
CUDA). Server parses and caches it, /health exposes as 'hwinfo'. Dashboard
renders it with icons at the top of the Runtime section so the user sees
immediately what the engine is running on.
* web: downgrade to React 18 — React 19 effect cleanup bug crashes the dashboard
React 19 treats the return value of every useEffect callback as a
cleanup function. In practice, any async interaction (streamChat's
rapid onDelta re-renders, the health poller, auto-connect) caused
React 19 to store a Promise as inst.destroy and crash with
'destroy_ is not a function' on the next commit cycle. The bug
reproduced in incognito, without StrictMode, and with every effect
converted to explicit block bodies returning undefined — it is a
React 19 regression in the passive-effect unmount path.
React 18 does not exhibit this behavior. Pin to React 18 until
React 19 is fixed or the codebase migrates to a pattern React 19
handles correctly. All 17 vitest pass, ErrorBoundary retained.
* web: Brain page — the expert cortex, live
A 76x256 canvas grid, one cell per expert (19,456 total): colour =
tier (green VRAM / blue RAM / grey disk), brightness = routing heat
(log-bucketed .coli_usage counts), and a white pulse that flashes on
every expert routed in the current turn and decays — you watch the
model think.
- Engine: EMAP protocol line (1 byte/expert: 2bit tier + 6bit heat,
hex) at READY and after each turn; HITS bitmap of this turn's routed
experts (set where eusage increments, cleared on emit).
- Server: parses both, GET /experts serves {rows, cols, map, hits, seq}.
- Web: Brain tab next to Chat; canvas renderer with rAF pulse decay;
hover tooltip shows layer/expert/tier/heat plus an honest depth-role
heuristic (early = surface features ... late = output shaping, MTP
row labelled as the speculative draft head).
* web: Brain page responsive — cells sized from container via ResizeObserver
Cell size derives from the wrapper's actual client box instead of fixed
1400x900, re-rendering on resize; mobile media query tightens padding,
legend and tooltip. The cortex now fills whatever screen it gets.
The engine opens/mmaps every safetensors shard of the model (144+ files
for GLM-5.2). On macOS the default soft fd limit is 256, so a stock
terminal session fails while loading with:
<model>/out-00125.safetensors: Too many open files
the engine exited while loading
Raise the soft limit toward the hard limit (capped at 65536) in the
coli launcher before spawning the engine, so users don't need a manual
'ulimit -n' in every shell. No-op on Windows and on shells whose limit
is already sufficient.
Co-authored-by: Harvad Lee <hongyanab@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Add make install/uninstall targets
Neither the root Makefile nor c/Makefile had an install target,
so 'build+install' never actually installed anything (fixes#164).
Adds standard PREFIX/DESTDIR/BINDIR-respecting install and uninstall
targets that place glm and coli in $(BINDIR).
* Split install: coli in bin/, engine + support files in libexec/
Addresses feedback on #164 from JustVugg and yurivict: coli goes to
$(BINDIR), glm/olmoe and their Python support modules
(resource_plan.py, doctor.py, openai_server.py, tools/) go to
$(LIBEXECDIR) (default $(PREFIX)/libexec/colibri), matching typical
Unix/FreeBSD-port conventions for a wrapper vs. its internals.
coli now resolves the engine path in this order:
1. $COLI_ENGINE if set (explicit override)
2. glm next to itself (run-in-place from a source checkout, unchanged)
3. $(LIBEXECDIR)/glm (installed layout), also added to sys.path so
the Python support modules still import correctly
Also adds a 'bench' target (builds iobench) since only cuda-bench
existed before.
Tested locally (WSL2/Ubuntu):
- run-in-place: cd c && python3 coli info -> engine ready
- installed: make install PREFIX=$HOME/.local && ~/.local/bin/coli info -> engine ready (found via libexec)
- make uninstall cleans both bin/ and libexec/colibri/ fully
Rebased onto current dev, split into 3 logical parts (all validated):
1. CPU portability (serve-mode _O_BINARY pipe fix — stock main hangs on MinGW without it; RAM detection cap 0->9/layer; POSIX guards for select/mmap/madvise; warmup script).
2. AVX-VNNI 128-bit int8/int4 dot kernel (Alder Lake+/Meteor Lake+), bit-identical to AVX2 (author-verified on Meteor Lake; compiles out to AVX2 elsewhere) + _mm256_extracti128_si256 typo fix that blocked -march=native.
3. CUDA DLL via LoadLibrary, gated behind CUDA_DLL=1 (host never links cudart; silent CPU fallback if absent; author-verified on RTX 5070 Ti).
Validated here: make check 59/59, oracle 32/32 TF, Windows cross-compile clean + glm.exe loads+runs via WSL interop. Fixes the #123 Windows build failure.
env_for() mapped --gpu/--vram only inside the --auto-tier branch, so the standalone form started a CPU-only engine with no warning (#121 — nearly published as a GPU benchmark). Now: --gpu none disables CUDA; --gpu list/auto sets COLI_CUDA/COLI_GPUS; --vram sets CUDA_EXPERT_GB; and --gpu/--vram on a CPU-only binary exit with a clear 'make glm CUDA=1' message instead of falling back silently. Tested on a CPU-only build: fail-fast fires for --gpu/--vram, --gpu none and default still start the CPU engine (no regression).
serve mode persists the compressed MLA KV-cache incrementally after every
turn (~182 KB/token appended, header count written last = crash-safe) and
resumes it at startup: the model remembers the whole conversation and zero
re-prefill happens. :reset and context-full restarts truncate the file.
The MTP layer's KV row is not saved; kv_start=-1 re-arms its decode window.
Validated: split-session answer byte-identical to an uninterrupted session
(tiny oracle, TEMP=0), and on the real 744B model a restarted chat resumed
58 tokens in 0.0s and recalled a fact from the previous session while
prefilling only the new question.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fenced code blocks become bordered boxes with the language label, **bold**
renders as real bold, `inline code` colored, # headers, - bullets. Works
char-by-char on the live stream (markers split across chunks are held back),
inline state resets per line, and orphan ``` fences right after a close
(a known int4 glitch) are swallowed. COLI_RAW=1 restores raw output.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Each architecture maps to its own engine binary (glm today; gptoss, qwenmoe
reserved). Registry in c/models.json (local, gitignored); chat shows a picker
when more than one model is installed. Dense models stay llama.cpp territory
- documented honestly in the README.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Faithful to the official modeling (transformers glm_moe_dsa): q from the
q_a latent via wq_b (32 heads x 128), k = LayerNorm(wk(h)) shared across
heads, interleaved RoPE on the first 64 dims, ReLU(q.k/sqrt(128)) weighted
by weights_proj(h)/sqrt(32), causal top-2048 per query.
- 'full' layers compute the selection (+ maintain the indexer k-cache from
token 0); 'shared' layers reuse it (IndexShare, index_topk_freq=4).
- Selection restricts both attention paths (absorbed decode + prefill
reconstruction). MTP row stays dense.
- Auto-detected like MTP: if out-idx-* weights are present for all full
layers, DSA arms itself; DSA=0 disables; DSA_FORCE/DSA_TOPK for testing.
- Validated on the tiny oracle (which ships indexer weights): selection
machinery forced on with keep=all keys reproduces dense attention exactly
(TF 32/32, gen 20/20); sparse smoke runs clean; kill switch verified.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Learning cache: expert usage persists in <SNAP>/.coli_usage across sessions
(atomic save every turn); at startup the hottest experts are auto-pinned in
RAM with half the expert budget (AUTOPIN=0 disables). The engine gets faster
the more you use it.
- Sampling: temperature + nucleus (official 1.0/0.95 defaults in chat; TEMP=0
= greedy). MTP/n-gram speculation stays lossless via rejection sampling
(accept draft w.p. p(draft); on reject resample with draft banned).
- coli: --temp flag.
- Converter: --indexer mode extracts DSA lightning-indexer weights
(resumable; needed for future sparse attention beyond 2048 ctx).
- pin_load/stats include the MTP row; usage histogram covers layer 78.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Vincenzo Fornaro