colibrì — tiny engine, immense model

**Tiny engine, immense model.** Run **GLM-5.2 (744B-parameter MoE)** on a consumer machine with ~25 GB of RAM β€” in pure C, with zero dependencies, by streaming experts from disk. ColibrΓ¬ is a lightweight, quality-preserving MoE runtime that treats VRAM, RAM, and storage as one managed memory hierarchy. Insufficient fast memory may reduce speed, but the default policy **never silently changes model precision or router semantics**. ``` $ ./coli chat 🐦 colibrΓ¬ v1.0 β€” GLM-5.2 Β· 744B MoE Β· int4 Β· streaming CPU βœ“ ready in 32s Β· resident 9.9 GB β€Ί ciao! β—† Ciao! 😊 Come posso aiutarti oggi? ``` ## See it running

colibrì web dashboard — live metrics, hardware panel, expert tiers

The web dashboard (./coli web): a 744B model at 4 tok/s, TTFT 1.6 s, disk 0 β€” full expert residency on 6Γ— RTX 5090, with live token metrics, the per-turn time breakdown, the VRAM/RAM/disk tier bar and the live mini-brain in the corner.

the Brain page β€” 19,456 experts as a live cortex

The Brain page: all 19,456 experts as a living cortex β€” colour is the storage tier, brightness is routing heat, and every expert routed in a turn flashes white. Hovering shows the expert's measured topic affinity.

the Atlas page β€” the measured expert atlas as a 3-D galaxy

The Atlas page: the measured expert atlas as a 3-D galaxy β€” 13,260 characterised experts, 1,041 replicated specialists clustering by topic (poetry, law, Chinese, SQL…). Position is measured routing affinity, not a learned embedding. Drag to spin.

## The idea A 744B Mixture-of-Experts model activates only ~40B parameters per token β€” and only ~11 GB of those change from token to token (the routed experts):

only ~5.4% of parameters are active per token

So the model doesn't need to *fit* in fast memory β€” it needs to be **placed**: - the **dense part** (attention, shared experts, embeddings β€” ~17B params) stays **resident in RAM at int4** (~9.9 GB); - the **19,456 routed experts** (75 MoE layers Γ— 256 + the MTP head, ~19 MB each at int4) live **on disk** (~370 GB) and are **streamed on demand**, with a per-layer LRU cache, a learned pinned hot-store, and an optional VRAM tier. The engine is a single C file (`c/glm.c`) plus small headers. No BLAS, no Python at runtime, no GPU required. ## How it works ### The per-token path

route β†’ union β†’ place β†’ overlap β†’ learn

Every layer of every token walks the same five steps. The design goal is that **placement only ever decides speed** β€” the router's decisions and the weights' precision are the same whether an expert answered from VRAM or from disk. ### One memory hierarchy instead of one memory requirement

VRAM / RAM / NVMe three-tier expert residency

The same engine spans the whole range: on a 25 GB laptop everything streams from disk (slow but correct); on a large host the entire expert set becomes resident (`CUDA_EXPERT_GB=auto PIN_GB=all`) and disk drops out of the decode path entirely. Between the tiers sits a **learning cache**: the engine records which experts *your* workload routes to (`.coli_usage`, updated every turn) and pins the hottest ones automatically β€” colibrΓ¬ literally gets faster the more you use it. On multi-socket hosts, `COLI_NUMA=1` interleaves the resident weights across memory controllers ([#82](https://github.com/JustVugg/colibri/issues/82)). ### Never wait for the disk twice Misses are expensive, so the engine spends most of its cleverness avoiding and overlapping them: each expert's three matrices are stored adjacent and read in one `pread`; a bounded async I/O pool (`PIPE=1`, default) loads missing experts while resident ones compute; batched positions read each unique expert once (**batch-union**); and a router-lookahead thread (`PILOT=1`) prefetches the next layer's experts β€” routing is measurably **71.6% predictable one layer ahead**. On GPUs, the resident pipeline (`COLI_CUDA_PIPE=2`) keeps the residual stream on-device across layers so the CPU expert loop runs uninterrupted; on Apple Silicon an experimental [Metal backend](docs/metal.md) does the batched expert math on the unified-memory GPU. ### Faithful model, compressed state The forward pass is validated **token-exact against a `transformers` oracle** (teacher-forcing 32/32). MLA attention stores a compressed KV state β€” 576 floats/token instead of 32,768 (**57Γ— smaller**) β€” and persists it across restarts (`.coli_kv`): conversations reopen warm with zero re-prefill, byte-identical to an uninterrupted session. DSA sparse attention (GLM-5.2's lightning indexer) is implemented faithfully and validated by forcing full-key selection to reproduce dense attention exactly. ### Speculative decoding, honestly GLM-5.2's native MTP head drafts tokens that the main model verifies in one batched forward β€” 2.2–2.8 tokens/forward when it pays. Two hard-won rules ship as defaults: the MTP head must be **int8** (int4 heads collapse to 0–4% acceptance, [#8](https://github.com/JustVugg/colibri/issues/8)), and draft and verify must compute **the same function** β€” `SPEC_PIN=1` pins both to one kernel family ([#163](https://github.com/JustVugg/colibri/issues/163) is the full forensic story). Grammar-forced drafts ([`GRAMMAR=file.gbnf`](docs/grammar-draft.md)) add ~free acceptance on constrained JSON output. Whether speculation is a net win depends on your cache temperature β€” measure, and use `DRAFT=0` when it doesn't pay. ## What it achieves

measured decode speed by hardware class

Same engine, same int4 container β€” the hardware only changes where the experts live. Highlights from the [full benchmark tables](docs/benchmarks.md): - **6Γ— RTX 5090, full residency:** 5.8–6.8 tok/s decode, TTFT ~13 s ([experiment log](docs/experiments/glm52-6x5090-2026-07-12.md)); - **128 GB CPU-only desktop:** ~1.8 tok/s warm ([#200](https://github.com/JustVugg/colibri/issues/200)); - **single RTX 5070 Ti laptop-class box:** 1.07 tok/s via the GPU-resident pipeline ([#273](https://github.com/JustVugg/colibri/issues/273)); - **25 GB dev box:** 0.05–0.1 tok/s cold β€” the proven floor where this project started, and still the honest baseline. Quality is measured, not assumed: the int4 container's quantization cost and the scale-granularity/rotation ablations live in [docs/benchmarks.md](docs/benchmarks.md#quality-benchmark) and [#108](https://github.com/JustVugg/colibri/issues/108)/[#81](https://github.com/JustVugg/colibri/issues/81). ## Get started ### 1. Get the model A pre-converted **GLM-5.2 int4** container is on Hugging Face β€” **use the version with the int8 MTP heads**: **https://huggingface.co/mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp** > ⚠️ The original mirror ships int4 MTP heads β†’ 0% draft acceptance > ([#8](https://github.com/JustVugg/colibri/issues/8)). Check yours: > `ls -l /out-mtp-*` β€” int8 (correct) is `3527131672 / 5366238584 / 1065950496`. Or convert from the FP8 source yourself β€” one resumable command that never needs the full 756 GB on disk at once: ```bash cd c && ./setup.sh # checks gcc/OpenMP, builds, self-tests ./coli convert --model /nvme/glm52_i4 # download+convert shard by shard (python, one-time) ``` ### 2. Run it ```bash COLI_MODEL=/nvme/glm52_i4 ./coli chat # RAM budget, cache and MTP auto-detected COLI_MODEL=/nvme/glm52_i4 ./coli plan # inspect the planned VRAM/RAM/disk placement COLI_MODEL=/nvme/glm52_i4 ./coli doctor # read-only readiness check ./coli web --model /nvme/glm52_i4 # API + web dashboard on one port ./coli serve --model /nvme/glm52_i4 # OpenAI-compatible API only ``` The engine at runtime is pure C β€” python is only used by the one-time converter and the optional API gateway. ### 3. Go deeper | topic | doc | |---|---| | Benchmarks, community datapoints, quality measurements | [docs/benchmarks.md](docs/benchmarks.md) | | Tuning knobs, policies, the learning cache, prefetch | [docs/tuning.md](docs/tuning.md) | | Windows 11 native build (+ CUDA DLL) | [docs/windows.md](docs/windows.md) | | CUDA backend, VRAM expert tier, full residency | [docs/cuda.md](docs/cuda.md) | | Apple Silicon Metal backend | [docs/metal.md](docs/metal.md) | | OpenAI-compatible API, KV slots, web dashboard | [docs/api.md](docs/api.md) | | Grammar-forced drafts (structured output) | [docs/grammar-draft.md](docs/grammar-draft.md) | | Environment variable inventory | [docs/ENVIRONMENT.md](docs/ENVIRONMENT.md) | ## Supporting the project colibrΓ¬ started as a one-person project on a 12-core laptop with 25 GB of RAM; today its numbers come from a community of real machines. If it's useful to you: - ⭐ star the repo and share it; - πŸ› open issues with benchmark numbers from your hardware β€” datapoints move this project more than anything else; - πŸ’¬ reach out via GitHub issues to sponsor development or donate hardware. ## Repo layout ``` Makefile root build/check entry point c/ β”œβ”€β”€ glm.c single-file GLM engine β”œβ”€β”€ st.h, tok.h, json.h runtime headers β”œβ”€β”€ backend_cuda.* optional CUDA tier β”œβ”€β”€ Makefile build and local checks β”œβ”€β”€ coli user-facing CLI β”œβ”€β”€ openai_server.py OpenAI-compatible HTTP gateway β”œβ”€β”€ setup.sh one-command local setup β”œβ”€β”€ tools/ offline conversion, fixtures and benchmarks β”œβ”€β”€ scripts/ long-running conversion helpers └── tests/ dependency-free C and Python tests web/ browser UI (pure OpenAI-API client) desktop/ Tauri v2 desktop shell wrapping the web UI docs/ reference docs, experiments, media ``` The runtime path intentionally stays flat and readable: `glm.c` plus its small headers. From the repository root, `make`, `make check`, and `make clean` delegate to the engine Makefile. ## Why "colibrΓ¬" The hummingbird weighs a few grams, hovers in place, and visits a thousand flowers a day. This engine keeps a 744-billion-parameter giant alive on hummingbird rations: 25 GB of RAM, twelve CPU cores, and a lot of disk patience. ## License Apache 2.0. GLM-5.2 weights are released by Z.ai under MIT.