Merge branch 't321' into trial321b

# Conflicts: # README.md
2026-07-16 21:11:23 +02:00
parent cead3d8bf3 7bc9f6e2e9
commit d45cfa268a
16 changed files with 732 additions and 593 deletions
@@ -4,10 +4,10 @@
 **Tiny engine, immense model.** Run **GLM-5.2 (744B-parameter MoE)** on a consumer machine with ~25 GB of RAM — in pure C, with zero dependencies, by streaming experts from disk.
-Colibrì is a lightweight, quality-preserving MoE runtime that treats VRAM,
+Colibrì is a lightweight, quality-preserving MoE runtime that treats VRAM, RAM,
-RAM, and storage as one managed memory hierarchy. Insufficient fast memory may
+and storage as one managed memory hierarchy. Insufficient fast memory may reduce
-reduce speed, but the default policy never silently changes model precision or
+speed, but the default policy **never silently changes model precision or router
-router semantics.
+semantics**.
 ```
 $ ./coli chat
@@ -17,14 +17,14 @@ $ ./coli chat
  ◆ Ciao! 😊 Come posso aiutarti oggi?
 ```
 ## See it running
 <p align="center">
  <img src="docs/media/colibri-dashboard.png" width="900" alt="colibrì web dashboard — live metrics, hardware panel, expert tiers">
 </p>
-<p align="center"><em>The web dashboard (<code>./coli web</code>): a 744B model answering at 4+ tok/s end-to-end on 6× RTX 5090 —
+<p align="center"><em>The web dashboard (<code>./coli web</code>): a 744B model at <strong>4 tok/s, TTFT 1.6 s, disk 0</strong> —
-with live token metrics, the hardware panel, and the VRAM/RAM/disk expert tiers.</em></p>
+full expert residency on 6× RTX 5090, with live token metrics, the per-turn time breakdown,
 the VRAM/RAM/disk tier bar and the live mini-brain in the corner.</em></p>
 <p align="center">
  <img src="docs/media/colibri-brain.png" width="900" alt="the Brain page — 19,456 experts as a live cortex">
@@ -33,619 +33,174 @@ with live token metrics, the hardware panel, and the VRAM/RAM/disk expert tiers.
 brightness is routing heat, and every expert routed in a turn flashes white. Hovering shows the expert's
 <a href="https://github.com/JustVugg/colibri/issues/175">measured topic affinity</a>.</em></p>
-## Contents
+<p align="center">
-
+  <img src="docs/media/colibri-atlas.png" width="900" alt="the Atlas page — the measured expert atlas as a 3-D galaxy">
- [The idea](#the-idea)
+</p>
- [See it running](#see-it-running)
+<p align="center"><em>The <strong>Atlas</strong> page: the <a href="https://github.com/JustVugg/colibri/issues/175">measured expert atlas</a>
- [What's implemented](#whats-implemented)
+as a 3-D galaxy — 13,260 characterised experts, 1,041 replicated specialists clustering by topic
- [Honest numbers](#honest-numbers-wsl2-12-cores-25-gb-ram-nvme-via-vhdx)
+(poetry, law, Chinese, SQL…). Position is measured routing affinity, not a learned embedding. Drag to spin.</em></p>
 - [Download the model](#download-the-model)
 - [Web dashboard](#web-dashboard)
 - [Got a better machine?](#got-a-better-machine-try-it--heres-what-to-expect)
 ## The idea
-A 744B Mixture-of-Experts model activates only ~40B parameters per token — and only ~11 GB of those change from token to token (the routed experts). So:
+A 744B Mixture-of-Experts model activates only ~40B parameters per token — and
 only ~11 GB of those change from token to token (the routed experts):
- the **dense part** (attention, shared experts, embeddings — ~17B params) stays **resident in RAM at int4** (~9.9 GB);
+<p align="center">
- the **19,456 routed experts** (75 MoE layers × 256 experts + the MTP head, ~19 MB each at int4) live **on disk** (~370 GB) and are **streamed on demand**, with a per-layer LRU cache, an optional pinned hot-store, and the OS page cache as a free L2.
+  <img src="docs/media/sparse.png" width="880" alt="only ~5.4% of parameters are active per token">
 </p>
-The engine is a single C file (`c/glm.c`) plus small headers. No BLAS, no Python at runtime, no GPU required (an opt-in CUDA tier for pinned experts exists — see below).
+So the model doesn't need to *fit* in fast memory — it needs to be **placed**:
-## What's implemented
+- the **dense part** (attention, shared experts, embeddings — ~17B params) stays
  **resident in RAM at int4** (~9.9 GB);
 - the **19,456 routed experts** (75 MoE layers × 256 + the MTP head, ~19 MB each
  at int4) live **on disk** (~370 GB) and are **streamed on demand**, with a
  per-layer LRU cache, a learned pinned hot-store, and an optional VRAM tier.
- **Faithful GLM-5.2 (`glm_moe_dsa`) forward** — validated token-exact against a `transformers` oracle (teacher-forcing 32/32, greedy 20/20 on a tiny-random model with the real architecture).
+The engine is a single C file (`c/glm.c`) plus small headers. No BLAS, no Python
- **MLA attention** (q/kv-LoRA, interleaved partial RoPE) with **compressed KV-cache**: 576 floats/token instead of 32,768 (57× smaller — GLM-5.2 has 64 heads and no GQA).
+at runtime, no GPU required.
 - **DeepSeek-V3-style sigmoid router** (noaux_tc, routed_scaling_factor), shared expert, first-3-dense layers.
 - **Native MTP speculative decoding** — GLM-5.2's own multi-token-prediction head (layer 78) drafts tokens that the main model verifies in one batched forward. **The head must be int8** (the converter does this by default): at int4 draft acceptance collapses to 0–4% and speculation never engages; at int8 it's 39–59% acceptance, **2.2–2.8 tokens/forward** (community-measured, [#8](https://github.com/JustVugg/colibri/issues/8)). Lossless *in exact arithmetic* — but **not byte-identical to non-speculative greedy in practice** ([#100](https://github.com/JustVugg/colibri/issues/100)). This isn't MTP-specific: colibrì's quantized integer kernels are shape-dependent, so any batched (S>1) or GPU forward rounds slightly differently from the single-token path, and int4 GLM-5.2 sits close enough to argmax ties that such a rounding change can flip a token. MTP, the CUDA expert tier, and batched prefill are three different ways to trip the same sensitivity (community-confirmed in #100: swapping only the kernel family forks greedy output on 3/5 prompts, with **zero speculation**). Every emitted token is still the argmax of a *valid* forward — the continuation stays correct — it just isn't the same stream. For byte-exact reproducibility: `DRAFT=0` (no speculation), plus `IDOT=0 COLI_CUDA=0` if you also want kernel-family/GPU independence. Under sampling, rejection sampling keeps the distribution correct. Honest caveat from the same measurement: on a **cold** cache each verified draft routes to extra experts (~660 → ~1100 expert-loads/token), so speculation can be a net *time* loss until the cache/pin warms up.
 - **Grammar-forced speculative drafts** (`GRAMMAR=file.gbnf`, [#48](https://github.com/JustVugg/colibri/issues/48)) — on constrained-output workloads (JSON/NDJSON, function calling, structured extraction) the grammar itself is a third draft source: wherever it admits exactly **one** legal byte (braces, quotes, key names, enum bodies), that forced span is tokenized and injected as pre-accepted drafts with ~1.0 acceptance — no draft head, no lookup table, and it engages even with the int4 MTP head from [#8](https://github.com/JustVugg/colibri/issues/8). It never constrains sampling: forced spans are verified in the same batch-union forward as any draft, so a wrong or out-of-sync grammar cannot change the output — worst case is rejected drafts, and an adaptive guard turns the source off below 50% acceptance. Byte-level GBNF subset (literals, char classes, `| ( ) ? * +`, comments); `GRAMMAR_DRAFT=n` caps the forced span per forward (default 24). Composes with `DRAFT`/MTP, which fill the free-text gaps between forced spans. Full reference — mechanism, measured A/Bs, when it pays, prior art: [docs/grammar-draft.md](docs/grammar-draft.md).
 - **True sampling** — temperature + nucleus, defaults tuned for int4 reality (0.7 / 0.90; the official 1.0 / 0.95 samples quantization noise from the tail).
 - **Integer-dot kernels** (Q8_0-style int8 activations, AVX2 `maddubs`): int8 matmuls 1.4–2.5× faster (119 GFLOP/s measured), int4 1.8× in batch — routing decided per shape by measurement (int4 single-row stays f32: it measured slower).
 - **MLA weight absorption** (DeepSeek trick) for decode: no per-token k/v reconstruction — the query absorbs `kv_b`, context is projected after attention. Validated exact: TF 32/32 and generation 20/20 with absorption forced everywhere.
 - **Async expert readahead**: while one block of experts is being multiplied, the kernel is already reading the next (`WILLNEED`).
 - **Quantization kernels**: int8 / packed int4 / packed int2, per-row scales, AVX2, dequant-on-use. Packing validated bit-identical to the int8 container.
 - **DSA sparse attention** — GLM-5.2's lightning indexer, faithful to the reference `glm_moe_dsa` modeling: per-layer top-2048 causal key selection (full/shared indexer layers), auto-detected from the `out-idx-*` weights (`--indexer` converter mode, ~189 MB extracted from the FP8 repo). Validated exact: forcing the selection to keep every key reproduces dense attention token-for-token. `DSA=0` disables, `DSA_TOPK` overrides.
 - **KV-cache persistence** — conversations reopen **warm** across engine restarts: serve mode appends the compressed MLA KV to `.coli_kv` after every turn (~182 KB/token, crash-safe) and resumes it at startup with zero re-prefill. Validated byte-identical to an uninterrupted session. `KVSAVE=0` disables.
 - **Router-lookahead prefetch** (`PILOT=1`, experimental) — the next layer's routing is 71.6% predictable from the current layer's post-attention state (measured); a dedicated I/O thread prefetches those experts while the current layer computes.
 - **Batch-union MoE**: in prefill (and MTP verification), each unique expert of the batch is read once and applied to every position that routes to it.
 - **Byte-level BPE tokenizer in C** (GPT-2-style with Unicode-property regex, 320k merges).
 - **RAM safety**: the expert cache is auto-sized from `MemAvailable` at startup — an honest peak projection (working set, KV, MTP row, reconstruction buffers) so the kernel OOM-killer never fires.
 - **Offline FP8→int4 converter** (`c/tools/convert_fp8_to_int4.py`): downloads one shard at a time (~5 GB), dequants (128×128 block scales), requantizes to the engine's container, deletes the shard — the 756 GB FP8 checkpoint never needs to exist on disk at once. Resumable.
-## Honest numbers (WSL2, 12 cores, 25 GB RAM, NVMe via VHDX)
+## How it works
-Detailed GPU experiment: [GLM-5.2 on 6x RTX 5090](docs/experiments/glm52-6x5090-2026-07-12.md) — full expert residency across VRAM+RAM reaches 6.84 tok/s single-request decode.
+### The per-token path
-| metric | value |
+<p align="center">
-|---|---|
+  <img src="docs/media/token-path.png" width="880" alt="route → union → place → overlap → learn">
-| model on disk (int4 container) | ~370 GB |
+</p>
 | resident RAM (dense, int4) | 9.9 GB |
 | load time | ~30 s |
 | peak RSS during chat | ~20 GB (auto-capped) |
 | cold decode cost | ~11 GB disk reads/token (75 layers × 8 experts) |
 | disk ceiling (this dev box's drive) | ~1 GB/s → ~0.05–0.1 tok/s cold |
 | MTP speculation (int8 head) | 2.2–2.8 tok/forward measured ([#8](https://github.com/JustVugg/colibri/issues/8)) |
-This is not fast. It is a 744B frontier-class model **answering correctly on a machine that costs less than one H100 fan**. Warm cache, pinned hot experts and MTP push the useful-response latency down considerably; the physics of the disk does the rest.
+Every layer of every token walks the same five steps. The design goal is that
 **placement only ever decides speed** — the router's decisions and the weights'
 precision are the same whether an expert answered from VRAM or from disk.
-### SSD note
+### One memory hierarchy instead of one memory requirement
 Cold starts are heavy on random reads (~11 GB/token), but reads don't meaningfully wear an SSD — colibrì's streaming is read-only. The real concerns under heavy use are (1) **swap traffic** if the system runs out of RAM (writes do wear the drive — keep a sane `--ram` budget; colibrì's auto-budget is designed to stay clear of swap) and (2) **sustained thermals**: hours at full read duty cycle will heat cheaper drives. Monitor drive temperature and health.
-## Download the model
+<p align="center">
  <img src="docs/media/tiers.png" width="880" alt="VRAM / RAM / NVMe three-tier expert residency">
 </p>
-A pre-converted **GLM-5.2 int4** model for colibrì is available on Hugging Face — **use the version with the int8 MTP heads** (matey-0's clone):
+The same engine spans the whole range: on a 25 GB laptop everything streams from
 disk (slow but correct); on a large host the entire expert set becomes resident
 (`CUDA_EXPERT_GB=auto PIN_GB=all`) and disk drops out of the decode path
 entirely. Between the tiers sits a **learning cache**: the engine records which
 experts *your* workload routes to (`.coli_usage`, updated every turn) and pins
 the hottest ones automatically — colibrì literally gets faster the more you use
 it. On multi-socket hosts, `COLI_NUMA=1` interleaves the resident weights across
 memory controllers ([#82](https://github.com/JustVugg/colibri/issues/82)).
 ### Never wait for the disk twice
 Misses are expensive, so the engine spends most of its cleverness avoiding and
 overlapping them: each expert's three matrices are stored adjacent and read in
 one `pread`; a bounded async I/O pool (`PIPE=1`, default) loads missing experts
 while resident ones compute; batched positions read each unique expert once
 (**batch-union**); and a router-lookahead thread (`PILOT=1`) prefetches the next
 layer's experts — routing is measurably **71.6% predictable one layer ahead**.
 On GPUs, the resident pipeline (`COLI_CUDA_PIPE=2`) keeps the residual stream
 on-device across layers so the CPU expert loop runs uninterrupted; on Apple
 Silicon an experimental [Metal backend](docs/metal.md) does the batched expert
 math on the unified-memory GPU.
 ### Faithful model, compressed state
 The forward pass is validated **token-exact against a `transformers` oracle**
 (teacher-forcing 32/32). MLA attention stores a compressed KV state — 576
 floats/token instead of 32,768 (**57× smaller**) — and persists it across
 restarts (`.coli_kv`): conversations reopen warm with zero re-prefill,
 byte-identical to an uninterrupted session. DSA sparse attention (GLM-5.2's
 lightning indexer) is implemented faithfully and validated by forcing full-key
 selection to reproduce dense attention exactly.
 ### Speculative decoding, honestly
 GLM-5.2's native MTP head drafts tokens that the main model verifies in one
 batched forward — 2.2–2.8 tokens/forward when it pays. Two hard-won rules ship
 as defaults: the MTP head must be **int8** (int4 heads collapse to 0–4%
 acceptance, [#8](https://github.com/JustVugg/colibri/issues/8)), and draft and
 verify must compute **the same function** — `SPEC_PIN=1` pins both to one
 kernel family ([#163](https://github.com/JustVugg/colibri/issues/163) is the
 full forensic story). Grammar-forced drafts
 ([`GRAMMAR=file.gbnf`](docs/grammar-draft.md)) add ~free acceptance on
 constrained JSON output. Whether speculation is a net win depends on your
 cache temperature — measure, and use `DRAFT=0` when it doesn't pay.
 ## What it achieves
 <p align="center">
  <img src="docs/media/ladder.png" width="880" alt="measured decode speed by hardware class">
 </p>
 Same engine, same int4 container — the hardware only changes where the experts
 live. Highlights from the [full benchmark tables](docs/benchmarks.md):
 - **6× RTX 5090, full residency:** 5.8–6.8 tok/s decode, TTFT ~13 s
  ([experiment log](docs/experiments/glm52-6x5090-2026-07-12.md));
 - **128 GB CPU-only desktop:** ~1.8 tok/s warm ([#200](https://github.com/JustVugg/colibri/issues/200));
 - **single RTX 5070 Ti laptop-class box:** 1.07 tok/s via the GPU-resident
  pipeline ([#273](https://github.com/JustVugg/colibri/issues/273));
 - **25 GB dev box:** 0.05–0.1 tok/s cold — the proven floor where this project
  started, and still the honest baseline.
 Quality is measured, not assumed: the int4 container's quantization cost and the
 scale-granularity/rotation ablations live in
 [docs/benchmarks.md](docs/benchmarks.md#quality-benchmark) and
 [#108](https://github.com/JustVugg/colibri/issues/108)/[#81](https://github.com/JustVugg/colibri/issues/81).
 ## Get started
 ### 1. Get the model
 A pre-converted **GLM-5.2 int4** container is on Hugging Face — **use the
 version with the int8 MTP heads**:
 **https://huggingface.co/mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp**
-> ⚠️ **The MTP head must be int8.** The original mirror ([jlnsrk/GLM-5.2-colibri-int4](https://huggingface.co/jlnsrk/GLM-5.2-colibri-int4)) ships **int4** MTP heads, which give **0% draft acceptance** — speculation silently never engages and you lose the ~2× MTP lever. This is the single most common "why is MTP stuck at 0%?" report ([#8](https://github.com/JustVugg/colibri/issues/8), [#102](https://github.com/JustVugg/colibri/issues/102)). The int8 head gives the measured **39–59% acceptance**. matey-0's clone above is the original int4 model with the three `out-mtp-*` files already swapped to int8 — download that one and you're done.
+> ⚠️ The original mirror ships int4 MTP heads → 0% draft acceptance
->
+> ([#8](https://github.com/JustVugg/colibri/issues/8)). Check yours:
-> Check what you have: `ls -l <model>/out-mtp-*`
+> `ls -l <model>/out-mtp-*` — int8 (correct) is `3527131672 / 5366238584 / 1065950496`.
 > · **int8 (correct):** `3527131672 / 5366238584 / 1065950496`
 > · **int4 (0% acceptance):** `1765523544 / 2686077736 / 536747200` — if you see these, replace just those three files from the int8 mirror.
-Download the repository and point `COLI_MODEL` to its directory:
+Or convert from the FP8 source yourself — one resumable command that never needs
 the full 756 GB on disk at once:
 ```bash
-COLI_MODEL=/path/to/GLM-5.2-colibri-int4-with-int8-mtp ./coli chat
+cd c && ./setup.sh                        # checks gcc/OpenMP, builds, self-tests
 ./coli convert --model /nvme/glm52_i4     # download+convert shard by shard (python, one-time)
 ```
-This skips the FP8 → int4 conversion step entirely. Thanks to DatPat for the original mirror and matey-0 for the int8-head clone.
+### 2. Run it
 ### Quick start
 ```bash
-cd c
+COLI_MODEL=/nvme/glm52_i4 ./coli chat     # RAM budget, cache and MTP auto-detected
-./setup.sh                      # checks gcc/OpenMP, builds, self-tests
+COLI_MODEL=/nvme/glm52_i4 ./coli plan     # inspect the planned VRAM/RAM/disk placement
-
+COLI_MODEL=/nvme/glm52_i4 ./coli doctor   # read-only readiness check
-# ONE command does everything model-side: downloads GLM-5.2-FP8 shard by shard
+./coli web  --model /nvme/glm52_i4        # API + web dashboard on one port
-# (never needs the full 756 GB at once), converts to the int4 container, then
+./coli serve --model /nvme/glm52_i4       # OpenAI-compatible API only
 # converts the MTP head for speculative decoding. Resumable at any point.
 # Conversion (only) needs python with: pip install torch safetensors huggingface_hub numpy
 ./coli convert --model /nvme/glm52_i4     # ~400 GB free on a real ext4/NVMe path
 # chat — RAM budget, expert cache and MTP are all detected automatically:
 COLI_MODEL=/nvme/glm52_i4 ./coli chat
 ```
-Inspect the planned storage hierarchy before loading the model:
+The engine at runtime is pure C — python is only used by the one-time converter
 and the optional API gateway.
-```bash
+### 3. Go deeper
 COLI_MODEL=/nvme/glm52_i4 ./coli plan
 COLI_MODEL=/nvme/glm52_i4 ./coli plan --gpu 0,1 --ram 128 --vram 48 --json
-# apply the bounded plan to the normal runner
+| topic | doc |
 COLI_MODEL=/nvme/glm52_i4 ./coli chat --auto-tier
 ```
 `coli plan` reads only safetensors headers and reports the model's exact dense/expert
 footprint, runtime RAM reserve, safe expert-cache cap, and bounded VRAM hot tier. Its
 versioned JSON output is intended to be shared by the CLI, API server, Web UI, and
 desktop shell; it does not allocate model tensors or start inference.
 `--auto-tier` applies the same plan to `chat`, `run`, `serve`, and benchmarks. It
 sets the RAM budget and context immediately; the VRAM tier is enabled only when
 the current `glm` binary is linked with CUDA. Explicit flags and environment
 variables keep precedence over automatic values.
 Before loading the model, `coli doctor` performs a read-only readiness check and
 explains whether the selected Disk/RAM/VRAM placement is runnable:
 ```bash
 COLI_MODEL=/nvme/glm52_i4 ./coli doctor
 COLI_MODEL=/nvme/glm52_i4 ./coli doctor --gpu 0 --ram 128 --json
 ```
 Doctor validates the model directory, config, tokenizer, safetensors headers,
 engine executable, available RAM, requested NVIDIA devices, CUDA linkage, and the
 same placement budget used by `coli plan`. It never starts `glm`, reads tensor
 payloads, imports a model framework, or creates a CUDA context. The versioned JSON
 report uses stable check IDs for automation. Warnings keep exit status 0; missing
 requirements or an unsafe RAM projection return 1, while invalid CLI values return 2.
 The engine at runtime is pure C — python is only used by the one-time converter.
 ### Windows 11 (native, no WSL)
 colibrì builds and runs natively on Windows 11 x86-64 with MinGW-w64. The port adds
 a `_WIN32` compatibility layer in `c/compat.h` that maps POSIX I/O to the Windows API
 (pread → ReadFile+OVERLAPPED, posix_fadvise no-op, aligned allocation, MoveFileEx rename,
 GlobalMemoryStatusEx RAM detection). All platform differences stay in `compat.h`; the
 engine source is unchanged.
 **Toolchain:** GCC via [winlibs](https://winlibs.com/) or MSYS2 MinGW-w64. Tested with
 GCC 16.1.0 (x86_64-ucrt-posix-seh).
 ```powershell
 # One-time toolchain install (pick one):
 scoop install mingw-winlibs                    # portable, no shell needed
 # or: pacman -S mingw-w64-x86_64-gcc make     # via MSYS2
 # Python (needed by the `coli` CLI and API server). A fresh Windows resolves
 # `python` to a Microsoft Store alias stub that opens the Store instead of
 # running anything (#198) — installing the real interpreter replaces the stub:
 winget install -e --id Python.Python.3.12
 # Build (from c/ directory):
 make glm.exe            # GLM-5.2 engine (static, no DLL dependencies)
 make olmoe.exe          # OLMoE engine (same shims)
 make iobench.exe        # disk I/O benchmark
 make test-c             # run C tests
 make test-python        # run Python tests (requires python)
 # AVX-VNNI: Intel Alder Lake+ (and Meteor Lake+) CPUs have a 128-bit int8
 # dot-product instruction (VPDPBUSD) the engine can use for ~1.3x faster
 # quantized matmul. The x86-64-v3 default (portable AVX2) compiles it out;
 # build for THIS machine to enable it:
 make glm.exe ARCH=native                       # banner prints "idot: avx-vnni"
 # Verify (tiny model, 2.4 MB):
 pip install torch transformers safetensors huggingface_hub
 python tools/make_glm_oracle.py                # generate tiny oracle
 SNAP=./glm_tiny TF=1 ./glm.exe 64 16 16        # expect "32/32 positions"
 # Run with real model:
 SNAP=D:\glm52_i4 ./glm.exe 64 4 16            # batch inference
 python coli chat --model D:\glm52_i4            # interactive chat
 python coli serve --model D:\glm52_i4            # OpenAI-compatible API
 ```
 **Warmup (overnight cache priming):** the engine's expert cache learns from
 your workload. The included `warmup.ps1` script runs `coli run` in a loop with
 diverse prompts to build the `.coli_usage` histogram unattended, so the next
 real session starts with a large, accurate hot-expert pin. Each run saves usage
 atomically on clean completion.
 ```powershell
 .\warmup.ps1 -Rounds 1 -Ngen 32               # ~60-90 min, durable progress
 ```
 **NVIDIA GPU (optional, via runtime DLL):** on Windows the engine is built with
 MinGW gcc but CUDA kernels require MSVC + nvcc. The split is clean: build the
 CUDA backend into a standalone `coli_cuda.dll` (nvcc + MSVC), then the host
 `glm.exe` loads it at runtime via `LoadLibrary` (`c/backend_loader.c`). The host
 never links cudart directly; if the DLL is absent the engine falls back to CPU
 without error.
 ```powershell
 # Prerequisites: CUDA Toolkit + MSVC Build Tools (cl.exe) + nvcc on PATH.
 # Build the DLL from a shell with the MSVC environment set (vcvars64.bat or
 # "x64 Native Tools Command Prompt for VS"):
 make cuda-dll CUDA_HOME="C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.8" CUDA_ARCH=sm_120
 # Build the host with the runtime loader (CUDA_DLL=1 adds -DCOLI_CUDA and
 # links backend_loader.o instead of cudart):
 make glm.exe CUDA_DLL=1 ARCH=native
 # Run with the GPU expert tier (8 GB VRAM budget here; scale to your free VRAM):
 $env:COLI_CUDA="1"; $env:COLI_GPU="0"; $env:CUDA_EXPERT_GB="8"
 python coli chat --model D:\glm52_i4 --topp 0.7
 ```
 The DLL exports 11 `extern "C"` symbols (`coli_cuda_init`, `coli_cuda_matmul`,
 etc.); `backend_loader.c` resolves them via `GetProcAddress` on first use.
 `ColiCudaTensor*` is opaque to the host (stored, never dereferenced), so the
 MSVC-allocated struct is safe across the ABI boundary. `CUDA_ARCH` must match
 your GPU's compute capability (e.g. `sm_120` for Blackwell / RTX 50-series,
 `sm_89` for Ada / RTX 40-series).
 **Status:** Phase 1 complete (compiles, correct, static-linked). The Windows
 GPU tier (runtime `coli_cuda.dll` via `LoadLibrary`) is implemented and
 verified on RTX 50-series (sm_120). O_DIRECT (Phase 2) and full-model
 validation against the transformers oracle remain separate workstreams.
 ### OpenAI-compatible API
 `coli serve` keeps one model process loaded and exposes a text-only OpenAI-compatible
 HTTP API. The gateway uses only the Python standard library; inference still runs in
 the same dependency-free C engine.
 ```bash
 cd c
 COLI_MODEL=/nvme/glm52_i4 COLI_API_KEY=local-secret ./coli serve \
  --host 127.0.0.1 --port 8000 --model-id glm-5.2-colibri
 curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Authorization: Bearer local-secret' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "glm-5.2-colibri",
    "messages": [{"role": "user", "content": "Hello"}],
    "stream": true
  }'
 ```
 Implemented endpoints are `GET /v1/models`, `GET /v1/models/{model}`,
 `POST /v1/chat/completions`, and legacy `POST /v1/completions`. Chat and
 completion requests support JSON responses, SSE streaming, usage counts,
 `max_tokens`/`max_completion_tokens`, `temperature`, and `top_p`. The extension
 `enable_thinking: true` enables GLM-5.2's reasoning block; the standard
 `reasoning_effort` field also enables it unless set to `none`.
 The first version is deliberately text-only and serves one generation at a time:
 the 744B model stays in one persistent process, so concurrent HTTP requests queue
 instead of loading duplicate model copies. Tools, image/audio input, custom stop
 sequences, log probabilities, and token penalties return an explicit error rather
 than being silently ignored. The default bind address is localhost; set
 `COLI_API_KEY` before exposing the server beyond the machine.
 Browser access from the Vite development server and Tauri local origins is enabled
 by default. Repeat `--cors-origin https://your-ui.example` to allow another exact
 origin, or use `--cors-origin '*'` only on a trusted local network.
 The engine owns one mutable KV context, so HTTP generation uses a bounded FIFO
 admission queue instead of pretending to run unsafe parallel sequences. Configure it
 with `--max-queue N` (default 8) and `--queue-timeout SECONDS` (default 300), or the
 `COLI_MAX_QUEUE` / `COLI_QUEUE_TIMEOUT` environment variables. Saturated and timed-out
 requests receive OpenAI-shaped HTTP 429 errors before streaming headers are sent.
 `GET /health` exposes active/queued/completed/rejected counters, and successful
 generation responses include `x-colibri-queue-wait-ms`.
 ### Isolated KV contexts
 `coli serve --kv-slots N` allocates up to 16 independent sequence contexts. Requests
 select one with the optional integer `cache_slot` field; ordinary OpenAI clients omit
 it and keep the original slot 0 behavior.
 ```json
 {
  "model": "glm-5.2-colibri",
  "messages": [{"role": "user", "content": "Continue this conversation"}],
  "cache_slot": 1
 }
 ```
 Each slot owns its token history, compressed MLA/DSA KV memory, MTP window, and
 crash-safe persistence file (`.coli_kv`, `.coli_kv.1`, ...). The engine still executes
 one sequence at a time; this establishes explicit KV ownership without pretending that
 threaded HTTP is continuous batching. RAM admission accounts for every configured slot.
 Use `COLI_KV_SLOTS=N` as the environment equivalent. Start with a small value: at the
 default 4096-token context, every slot costs hundreds of MB.
 ### Experimental Metal backend (Apple Silicon)
 On Apple Silicon the decode profile is matmul-bound, and unified memory removes the
 PCIe copy tax that keeps CUDA's streaming experts on the CPU — so colibrì has an
 opt-in Metal backend that runs the **routed-expert SwiGLU (batched, zero-copy from
 the RAM slabs)**, the **fused decode attention** (full MLA layer in one command
 buffer, S≤4), and **prefill's large GEMMs** on the GPU. Token-exact vs the CPU path.
 ```bash
 cd c
 make glm METAL=1          # macOS only; no Xcode needed (shader compiles at runtime)
 make metal-test           # standalone kernel/attention correctness vs CPU reference
 COLI_METAL=1 COLI_MODEL=/path/glm52_i4 ./coli chat --ram 96
 ```
 Measured on an M4 Max (128 GB, warm cache, MTP on): CPU 0.30 → Metal **0.42 tok/s (~1.4×)**
 (best config adds `DIRECT=1`; ~3× vs this machine's first cold run).
 Key design points: Metal's ~5 ms submit latency makes per-matmul dispatch a loss —
 everything is batched into few command buffers per layer, and the resident experts'
 GPU work is submitted *before* the missed experts' disk reads so I/O and compute
 overlap. `COLI_METAL_GEMM_MIN` tunes the prefill GEMM row threshold (default 16).
 Streaming, cache, MTP, DSA and the persistence formats are unchanged; every GPU
 path falls back to the CPU per-block on any fault. Numerics are dequant→f32-MAC
 (same as the CUDA tier); greedy outputs are byte-identical to the CPU engine.
 ### Experimental resident CUDA backend
 colibrì includes an opt-in CUDA backend for model-resident tensors. Streaming
 experts deliberately remain on the original CPU path for now: copying an expert
 from NVMe to the GPU on every use would only replace the disk bottleneck with a
 PCIe bottleneck. Resident quantized tensors are uploaded lazily once and reused.
 ```bash
 cd c
 make cuda-test CUDA=1                  # q8/q4/q2/f32 kernel correctness
 make CUDA=1
 # optional dense-path experiment (hot experts are configured below)
 COLI_CUDA=1 COLI_GPU=0 CUDA_DENSE=1 SNAP=/nvme/glm52_i4 ./glm 64 4 4
 ```
 Requirements: Linux, an NVIDIA driver, and a CUDA Toolkit under
 `/usr/local/cuda` (override with `CUDA_HOME=/path/to/cuda`). `CUDA_ARCH=native`
 builds for the GPU in the current machine; set an explicit architecture when
 cross-compiling. Requesting CUDA with a CPU-only binary, an invalid device, or
 an unavailable runtime fails at startup instead of silently falling back.
 The normal `make` build and runtime behavior are unchanged. CUDA defaults to an
 expert-only accelerator. `CUDA_DENSE=1` additionally distributes resident
 dense/attention projection tensors round-robin across the selected devices;
 their projected footprint is reserved before the expert tier is placed. On six
 RTX 5090s with a 150 GB expert tier, a warmed two-request/64-token GLM-5.2 run
 improved from 1.650 to 2.157 aggregate tok/s (+30.8%) while retaining the full
 expert tier. Treat this as an opt-in until the projected dense set and the 2 GB
 per-device runtime reserve fit the target GPUs.
 A measured `PIN` profile can promote its hottest experts into the persistent
 VRAM tier while keeping the rest in RAM:
 ```bash
 STATS=stats.txt SNAP=/nvme/glm52_i4 ./glm 64 4 4   # collect routing frequencies first
 COLI_CUDA=1 COLI_GPU=0 CUDA_EXPERT_GB=16 \
 PIN=stats.txt PIN_GB=160 SNAP=/nvme/glm52_i4 ./glm 64 4 4
 # multi-GPU expert tier, 150 GB total budget across six 32 GB devices
 COLI_CUDA=1 COLI_GPUS=0,1,2,3,4,5 CUDA_EXPERT_GB=150 \
 CUDA_DENSE=1 PIN=stats.txt PIN_GB=300 RAM_GB=226 \
 SNAP=/nvme/glm52_i4 ./glm 64 4 4
 # large-RAM host: fill safe VRAM, then keep every remaining expert in RAM
 COLI_CUDA=1 COLI_GPUS=0,1,2,3,4,5 CUDA_EXPERT_GB=auto \
 CUDA_DENSE=1 COLI_CUDA_ATTN=1 PIN=stats.txt PIN_GB=all RAM_GB=auto \
 SNAP=/nvme/glm52_i4 ./glm 64 4 4
 ```
 Selected experts are uploaded during startup, so capacity failures occur before
 inference and the log reports their exact tensor footprint. The budget is clamped
 against free VRAM after reserving the projected dense resident set and 2 GB of
 runtime headroom per selected device. With `COLI_GPUS`, `CUDA_EXPERT_GB` is a
 total budget across the device set; experts are assigned whole to the
 least-loaded device that can hold them. Multi-GPU runs also default to
 `PIN_FILL=1`: the measured hot set is placed first, then unused VRAM is filled
 with zero-heat experts. `CUDA_RELEASE_HOST=1` (the multi-GPU default) releases
 the RAM copy after a successful upload and reloads it from disk only if CUDA
 later fails. Set either variable to `0` to restore the conservative behavior.
 When host backing is released, placement is disjoint and staged: the hottest
 prefix is loaded, uploaded to VRAM, and freed before the next-ranked suffix is
 loaded into RAM. `PIN_GB` therefore describes the combined ranked set rather
 than duplicate RAM and VRAM copies. On a 256 GB dual-socket host, moving from a
 150 GB VRAM + 130 GB RAM placement to 150 GB VRAM + 150 GB RAM raised fixed-token
 replay from 1.87 to 2.16 tok/s (+15.7%), reduced expert disk wait from 5.144s to
 3.948s, and kept the projected RAM peak below `RAM_GB=226`. The cache cap adjusts
 down automatically (54 to 40 in that run) so the larger pinned tier does not exceed
 the process budget. Start lower on hosts with less available RAM.
 `CUDA_EXPERT_GB=auto` fills each selected device only up to its measured free
 memory minus projected dense tensors and 2 GB of runtime headroom. `PIN_GB=all`
 then loads every remaining routed expert into RAM, eliminating decode-time disk
 misses when the host budget permits it. The regular `RAM_GB` guard still clamps
 the per-layer working cache and rejects unsafe projections; this mode is intended
 for dedicated high-memory inference hosts, not desktops running other workloads.
 On a dedicated 251 GiB host with six RTX 5090s, this mode selected a 176.7 GB
 VRAM expert tier and a 191.3 GB RAM tier (all 19,456 experts resident). The
 mode also adapts the VRAM tier every 16 emitted tokens by swapping hot RAM
 experts into existing GPU slots. A real 64-token greedy GLM-5.2 generation
 measured **6.00 tok/s decode**, up from
 2.20 tok/s end-to-end with the earlier 150 GB tier; expert hit rate was 100%
 and disk wait was zero. Prompt prefill is reported separately. This is a
 host-specific capacity result, not a portable default.
 Text-mode timing reports prefill separately from decode. The decode rate starts
 after the prompt KV is built, so it is comparable to `REPLAY` throughput without
 hiding time-to-first-token.
 MTP speculation defaults off on CUDA because cold draft routes increase expert
 traffic; an explicit `DRAFT=n` still overrides the default.
 On six RTX 5090 32 GB cards with GLM-5.2 int4, a 150 GB hot-first tier sustained
 0.94 token/s over a 64-token varied prompt (87.8% expert hit rate), and reached
 1.64 token/s on a warmed short prompt (99.3% hit rate). The same capacity filled
 without routing heat managed only 0.29 token/s, so profile quality matters more
 than raw VRAM capacity. These are single-run engineering measurements, not a
 portable performance guarantee.
 Current limitations: devices use independent contexts and synchronous
 host-staged activation copies—there is no P2P/NCCL dependency yet. Independent
 expert groups execute concurrently across devices, but a single expert is not
 sharded. The kernels are correctness-first custom kernels rather than
 cuBLAS/Tensor Core kernels.
 For a reproducible backend A/B without the full checkpoint, generate the
 deterministic 313M-parameter `glm_moe_dsa` fixture and run fixed-token replay:
 ```bash
 cd c
 python tools/make_glm_bench_model.py --output /nvme/colibri-bench-medium --device cuda
 python tools/benchmark_cuda_fixture.py --model /nvme/colibri-bench-medium --gpu 0
 ```
 The fixture has random weights and is not a language model. It exists only to
 preserve the real MLA/MoE/streaming shapes and compare CPU streaming, dense-only
 CUDA, CPU hot-store, and CUDA hot-expert execution with identical replay tokens.
 ### Web interface
 `web/` contains a community-contributed browser UI (React + TypeScript, a pure
 API client — it never touches the engine directly):
 ```bash
 cd web
 npm ci && npm run dev        # then point it at an OpenAI-compatible endpoint
 ```
 It speaks the standard OpenAI Chat Completions protocol with SSE streaming, so it
 works against the colibrì OpenAI-compatible server (in review, #21) or any other
 compatible endpoint. Nothing leaves the endpoint you configure. The terminal
 `coli chat` remains the first-class interface.
 Useful knobs (env or flags): `--temp T` token sampling temperature (default 0.7 + nucleus 0.90 — tuned for int4; 0 = greedy), `--topp 0.7` adaptive expert top-p (30–40% less disk), `--ngen N` max tokens per answer (`:more` in chat continues a truncated one), `--repin N` adapt RAM/VRAM hot experts every N emitted tokens, `AUTOPIN=0` disable the learning cache's auto-pin, `THINK=1` enable GLM-5.2's reasoning block, `DRAFT=n` MTP draft depth, `GRAMMAR=g.gbnf` grammar-forced drafts for constrained JSON/NDJSON output (`GRAMMAR_DRAFT=n` caps the forced span), `TF=1` teacher-forcing validation, `PILOT=1` router-lookahead disk prefetch (experimental — see below), `URING=1` Linux-only batched expert I/O (implies `PIPE=1`; also batches `PILOT_REAL`), `PIPE=0` disable the async expert-load pool (**default ON on Windows** — overlaps expert `pread` with the matmul so the CPU isn't idle waiting on the SSD; measured −18% disk service time), `RAM_GB=<n>` claim more RAM for the expert cache than the conservative auto-detect (e.g. `RAM_GB=31` on a 32 GB host raises the cache cap and hit rate measurably), `CAP_RAISE=0` don't auto-grow the expert cache.
 ### Resource policy
 `coli plan` reports the planned hot (VRAM), warm (RAM), and cold backing
 (disk) tiers, the reason for each placement, and the expected bottleneck. The
 default `--policy quality` and `--policy balanced` modes preserve checkpoint
 quantization and router decisions unless `--topk` or `--topp` is passed; those
 explicit lossy overrides print a warning and proceed.
 Auto-tier plans size OpenMP from physical cores and bind workers across cores.
 Memory-bound quantized kernels can regress sharply when SMT siblings compete
 for limited memory channels; explicit `OMP_*` settings always take precedence.
 ```bash
 coli plan --model /models/glm52_i4 --policy quality
 coli run --auto-tier --policy quality "Explain MoE offloading"
 # Explicit research-only router reduction:
 coli run --policy experimental-fast --topk 4 "Benchmark prompt"
 ```
 Disk is an immutable recovery source, not a normal decode target. If the plan
 leaves cold expert bytes on disk, speed depends on cache hit rate; output
 quality does not.
 Cold expert reads can use a deferred pipeline: resident RAM/VRAM experts execute
 while missing experts are loaded in a bounded background I/O pool, then the
 cold results join before the layer completes. The pool engages only under
 `PIPE=1`; `PIPE_WORKERS=n` sets its worker count (default 8). Profiling reports
 both disk service time and the smaller foreground-visible wait time so overlap
 is explicit rather than credited as unexplained speedup.
 `--policy balanced` enables lossless live placement (`REPIN=64`). At safe
 request boundaries, a per-layer LFRU score combines decaying session frequency
 with recent access and replaces at most four sufficiently colder pinned
 experts. `--policy quality` leaves live replacement off by default; `REPIN=0`
 always disables it. Persistent `.coli_usage` history and session-local LFRU
 state remain separate.
 For single-token q4 CPU experts, gate and up projections share one OpenMP
 dispatch while retaining the same per-row AVX2/NEON arithmetic. This removes
 one thread-team launch per RAM expert without activation requantization or a
 lower-precision fallback. It is a stepping stone toward a persistent native
 CPU expert pool, not a replacement for one.
 **The expert cache auto-sizes to your RAM** (since 2026-07-10): the engine now *raises* the LRU cap to fill your `--ram` budget instead of only lowering it. Before this fix a 128 GB machine ran with the same 8-experts/layer cache as a 16 GB one (issue #12) — **if you benchmarked colibrì before this date, rerun: your numbers were capped.**
 **Router-lookahead prefetch** (`PILOT=1`, experimental): GLM-5.2's expert routing is measurably predictable *ahead of time* — applying layer L+1's router to layer L's post-attention state recalls **71.6%** of the true top-8 (vs 41.3% for "same experts as last token"). `PILOT=1` uses this to issue next-layer expert readahead from a dedicated I/O thread while the current layer computes. On our dev box the disk is already ~80% saturated, so it measures neutral; on machines where compute and disk are balanced (like the Ryzen AI 9 in issue #12: 43% disk / 46% matmul) it should overlap real work — measurements welcome.
 **The learning cache**: the engine records which experts your usage actually routes to (`.coli_usage` next to the model, updated every turn) and at startup automatically pins the hottest ones in spare RAM. colibrì literally gets faster the more you use it.
 **Live tier adaptation** (`--repin N`, opt-in): at safe turn boundaries, a decaying
 session heat map replaces cold pinned experts with hotter streamed experts. Replacement
 loads the expert from disk into the existing RAM slot; GPU-backed slots immediately
 refresh the same VRAM tier budget. A 25% hysteresis and a four-swap limit prevent tier
 thrashing. Persistent `.coli_usage` remains the long-term signal and is not decayed.
 **Conversations reopen warm** (`.coli_kv`, since 2026-07-10): `coli chat` persists the compressed MLA KV-cache to disk after every turn (~182 KB/token, appended incrementally, crash-safe). Close the chat, reopen it tomorrow — the model still remembers the whole conversation and **zero re-prefill happens**: validated byte-identical to an uninterrupted session. `:reset` clears it, `KVSAVE=0` disables it.
 ## Web dashboard
 One command serves the OpenAI-compatible API **and** the web console on the same port, then opens your browser when the engine is ready:
 ```bash
 cd web && npm install && npm run build   # once
 ./coli web --model <model-dir>
 ```
 What you get:
 - **Chat** with live metrics: a flashing token counter while generating, then tok/s, time-to-first-token, prompt→completion counts and queue wait;
 - **Runtime panel**: your hardware (CPU, GPUs + VRAM, RAM, cores), the scheduler, and the live expert-tier bar — how many of the 19,456 experts sit in VRAM / RAM / disk right now;
 - **Brain**: the whole model as a 76×256 cortex, one cell per expert. Colour = tier, brightness = routing heat, and the experts routed in each turn flash white and decay — you watch the model think. Hover any cell for its tier, heat and [measured topic affinity](https://github.com/JustVugg/colibri/issues/175) (specialists for code, Chinese, math, law… live in layers 11–22);
 - **Profiling**: where each turn's wall time went — I/O wait vs expert matmul vs attention vs LM head — as stacked per-turn bars, plus throughput history, tokens-per-forward batching, and a table of the recent turns. The same phase timers behind the `PROFILE` line, streamed live.
 The dashboard talks to the engine over a few tiny protocol lines (`TIERS`, `EMAP`/`HITS`, `PROF`) and plain JSON endpoints — nothing heavier than the engine itself.
 ## Got a better machine? Try it — here's what to expect
 colibrì was built on deliberately humble hardware (12 cores, 25 GB RAM, an older DRAM-less NVMe behind a WSL2 VHDX that measured ~1 GB/s random on *this* drive — note WSL2 VHDX is not inherently slow: a community 5090 box measured 10.5 GB/s O_DIRECT through one, [#101](https://github.com/JustVugg/colibri/issues/101)). **Every one of those constraints is a knob your machine can turn up.** The engine needs: Linux (or WSL2), macOS, or **Windows 11 natively (MinGW-w64)**; gcc with OpenMP, AVX2, ≥16 GB RAM, and the ~370 GB int4 model on a local NVMe (ext4/NTFS — never a network/9p mount).
 **How to test it, in order:**
 ```bash
 cd c && ./setup.sh                 # build + architecture self-test (expects 32/32)
 # 1) measure YOUR disk the way the engine uses it (parallel 19 MB random reads):
 gcc -O2 -fopenmp iobench.c -o iobench
 ./iobench /path/to/glm52_i4/out-00069.safetensors 19 64 8 0   # buffered, 8 threads
 ./iobench /path/to/glm52_i4/out-00069.safetensors 19 64 8 1   # O_DIRECT (bypass cache)
 # Caveat (#86): iobench reads a bounded ~1 GB shard, so buffered reads on a big-RAM box
 # report the PAGE CACHE, not the disk. Use the O_DIRECT run (arg 1) for a true number, and
 # run it on a shard you haven't touched this session (a prior buffered run caches its pages).
 # On macOS there is no O_DIRECT — iobench uses F_NOCACHE, which stops *new* caching but can't
 # evict pages a prior buffered run already resident-mapped, so a macOS "O_DIRECT" figure right
 # after a buffered run still reads cache. Reboot or use a fresh shard for a real cold read.
 # 2) chat; watch the per-turn stats line (tok/s, expert hit-rate, RSS):
 COLI_MODEL=/path/to/glm52_i4 ./coli chat
 # 3) record expert usage, then pin the hottest experts in your spare RAM:
 STATS=stats.txt ./coli chat
 PIN=stats.txt PIN_GB=20 ./coli chat        # scale PIN_GB to your free RAM
 # 4) quality benchmarks (MMLU/HellaSwag/ARC):
 ./coli bench
 ```
 **Back-of-envelope predictions** (decode is disk-bound: a cold token costs ~11.4 GB of expert reads; MTP speculation roughly halves the effective cost *once the cache is warm*; RAM turns cold reads into free cache hits):
 | machine | expected |
 |---|---|
-| this dev box (WSL2 VHDX, ~1 GB/s, 25 GB RAM) | ~0.05–0.1 tok/s cold — proven baseline |
+| Benchmarks, community datapoints, quality measurements | [docs/benchmarks.md](docs/benchmarks.md) |
-| native Linux, PCIe4 NVMe (~3–5 GB/s random), 32 GB | ~0.5–1 tok/s |
+| Tuning knobs, policies, the learning cache, prefetch | [docs/tuning.md](docs/tuning.md) |
-| PCIe5 NVMe or 2×NVMe RAID0 (~8–12 GB/s), 64 GB (PIN ~40 GB of hot experts) | ~2–4 tok/s |
+| Windows 11 native build (+ CUDA DLL) | [docs/windows.md](docs/windows.md) |
-| 128–256 GB RAM, 12 cores (hot experts cached) | ~2–4 tok/s — matmul-bound: ~80 GFLOP/token vs ~250 GFLOP/s of our AVX2 kernels |
+| CUDA backend, VRAM expert tier, full residency | [docs/cuda.md](docs/cuda.md) |
-| same RAM + 24–32 cores, or AVX-512/VNNI kernels | ~5–15 tok/s — interactive; kernel work is the multiplier |
+| Apple Silicon Metal backend | [docs/metal.md](docs/metal.md) |
-
+| OpenAI-compatible API, KV slots, web dashboard | [docs/api.md](docs/api.md) |
-These are estimates, not measurements — if you run colibrì on serious hardware, **please open an issue with your numbers**: real datapoints from better machines are exactly what this project needs next.
+| Grammar-forced drafts (structured output) | [docs/grammar-draft.md](docs/grammar-draft.md) |
-
+| Environment variable inventory | [docs/ENVIRONMENT.md](docs/ENVIRONMENT.md) |
 ### Community benchmarks (measured)
 Real numbers from real machines, stock build (`setup.sh`, gcc 13), greedy decoding, `--ngen 32`, MTP active:
 | machine | disk (iobench, 19 MB × 64, 8 threads) | config | measured |
 |---|---|---|---|
 | Intel Core Ultra 7 270K Plus (24 threads) · WSL2 · 24 GB RAM · NVMe VHDX ([#2](https://github.com/JustVugg/colibri/issues/2)) | 1.96 GB/s buffered · 2.74 GB/s O_DIRECT | default | 0.07 tok/s · expert hit 3–4% · RSS 14.1 GB |
 | 〃 | 〃 | `--topp 0.7` | **0.11 tok/s** · expert hit 11% · RSS 14.7 GB |
 | Apple M5 Max (18 cores) · macOS · 128 GB unified · internal SSD ([#4](https://github.com/JustVugg/colibri/issues/4), [#5](https://github.com/JustVugg/colibri/issues/5)) | ~4 GB/s cold (the 14.2 GB/s reading was cache-influenced — see note) | default, MTP off | **1.06 tok/s** · expert hit 23% · RSS 21.8 GB |
 | Apple M5 Max · macOS · 128 GB unified · 2 TB SSD · **Metal backend** ([#72](https://github.com/JustVugg/colibri/pull/72), [#87](https://github.com/JustVugg/colibri/issues/87)) | (macOS O_DIRECT figure unreliable — see note) | Metal on · `--ram 96` · 39.7 GB warm pin · MTP off | **1.83 tok/s** · expert hit 66% · warmed 1.11 → 1.83 over the run |
 | 〃 · 46.9 GB pin (2.94M-selection history) · `--ram 110`, 1024-token run ([#103](https://github.com/JustVugg/colibri/issues/103)) | 〃 | Metal on (experts + attention) · MTP off | **2.06 tok/s** · hit 72.5% · coherent output · fastest datapoint yet (still on the pre-rebase Metal branch) |
 | Mac Mini M4 Pro · macOS · **48 GB** unified · **Metal backend** ([#107](https://github.com/JustVugg/colibri/issues/107)) | 6.59 GB/s F_NOCACHE (fresh shard) | Metal on · `--ram 38` | **0.30 tok/s** (vs 0.18 CPU-only) — entry Apple Silicon on a third the RAM beats the 32-core 9950X row |
 | Epyc 9654 ES · Linux · 4x16GB DDR5-4800-rdimm · Samsung PCIe Gen3 x4 NVME SSD | — | `MTP=1 DIRECT=1` | 0.31 tok/s · expert hit 35% · RSS 21.52 GB |
 | Ryzen AI 9 HX 370 (Framework 13) · Arch Linux · 128 GB · WD SN850X, BTRFS zstd ([#12](https://github.com/JustVugg/colibri/issues/12)) | — | int8 MTP head · `--cap 32` · 46.7 GB auto-learned PIN | **0.37 tok/s** · expert hit 66% · MTP acceptance 52% (2.59 tok/fw) · RSS 105 GB |
 | Ryzen 9 9950X (32 threads) · Linux · 123 GB · Crucial P3 QLC Gen3 ([#31](https://github.com/JustVugg/colibri/issues/31)) | 1.51 GB/s buffered | default, 2 runs from cold | 0.10 tok/s · hit 53% · profile 66% disk |
 | 〃 same machine, model moved to a Samsung 9100 PRO PCIe 5.0 ([#31](https://github.com/JustVugg/colibri/issues/31)) | **8.81 GB/s** O_DIRECT | 〃 (usage history retained) | **0.28 tok/s** · hit 57% · profile flips: 32% disk / **57% matmul** |
 | Ryzen AI Max+ 395 (Framework Desktop) · Ubuntu · 128 GB LPDDR5x · Intel Optane 905p PCIe 3.0 ([#39](https://github.com/JustVugg/colibri/issues/39)) | 3.27 GB/s buffered | int8 MTP head · fresh history (pure LRU, auto-raised cap 65) | 0.16 tok/s · hit 57% · profile 49% disk / 47% matmul |
 | 〃 five runs later — learned pin 47.6 GB ([#39](https://github.com/JustVugg/colibri/issues/39)) | 〃 | `--temp 0.7 --topp 0.7` | **0.40 tok/s** · hit 71% · fastest non-Apple datapoint |
 | Ryzen 7 9800X3D (16T) · WSL2 · 70 GB RAM · Samsung 9100 PRO PCIe 5.0 · RTX 5090 ([#101](https://github.com/JustVugg/colibri/issues/101)) | **10.51 GB/s** O_DIRECT | MTP off · learned pin 24 GB · hit 54% · OMP hot-team on | **0.41 tok/s** · disk-bound (36.5 s disk vs 24.0 s matmul) · **CUDA expert tier ≈ 0%** (AVX-512 CPU matches the 5090) · `--topp 0.7` → **0.52 tok/s** |
 | EPYC 7443 (24C/48T, Zen3 AVX2) · Linux · **430 GB RAM** · NVMe RAID-Z1 via TrueNAS VM ([#104](https://github.com/JustVugg/colibri/issues/104)) | ~1 GB/s (VM overhead) | 77.5 GB pin · cap auto-raised to 194/layer · MTP off | **1.00 tok/s** · **hit 98%** · disk eliminated → **RAM-bandwidth + matmul bound** (no AVX-512/VNNI on Zen3) |
 | Intel i5-12600K (10C/16T, AVX2) · **native Windows 11, no WSL** · 32 GB · MinGW GCC 16.1 ([#113](https://github.com/JustVugg/colibri/issues/113)) | buffered (no O_DIRECT on MinGW) | int8 MTP head · cold, small-RAM (cap ~2/layer) | **0.08 tok/s** · hit 3.7% · **MTP 57% acceptance** — first native-Windows datapoint, port validated |
 | Ryzen 9 9950X3D2 (16C/32T, avx512-vnni) · native Linux · 121 GB · Samsung 9100 PRO **PCIe Gen5** · RTX 5090 (28 GB expert tier, 1475 pinned) ([#120](https://github.com/JustVugg/colibri/issues/120)) | **11.48 GB/s** O_DIRECT | `MTP=0 DIRECT=1 PIPE_WORKERS=16 PREFETCH=1` | **1.23 tok/s** · MTP-off wins disk-bound · fastest x86 datapoint yet |
 | Ryzen AI Max+ 395 (Strix Halo, 16C/32T Zen5, avx512-vnni) · Arch Linux · 128 GB unified LPDDR5x · SK hynix P41 PCIe 4.0 ([#124](https://github.com/JustVugg/colibri/issues/124)) | — | `DIRECT=1 PIPE=1 --topp 0.7` · auto-pin | 0.06 cold → **1.10 tok/s** sustained · first Strix Halo / gfx1151 datapoint (unified memory: no discrete VRAM tier) |
 | Intel Core Ultra 9 185H (16C/22T, avx-vnni) · **native Windows 11, no WSL** · 32 GB · Crucial P3 QLC NTFS · RTX 5070 Ti (unused) ([#128](https://github.com/JustVugg/colibri/issues/128)) | — | int8 MTP head · **with [#131](https://github.com/JustVugg/colibri/pull/131) (pipe + RAM fixes), warm cache, no GPU** | 0.03 cold → **0.5 tok/s** warm (~7-prompt warmup) · cache-warming on native Windows once the portability blockers are fixed — stock main hung on the `\r\n` READY sentinel before #131 |
 | Dell Pro Max GB10 (DGX Spark: Grace 10×X925 + 10×A725, **aarch64 i8mm/sve2**) · Linux · 121 GB unified LPDDR5x · Dell OEM 4 TB NVMe · GB10 sm_121 ([#136](https://github.com/JustVugg/colibri/issues/136)) | **5.58 GB/s** O_DIRECT (NVIDIA-OEM unit in #76 was 10.74 — same platform, different SSD) | int8 MTP head · warm cache | 0.21 cold → **0.50 tok/s** warm · hit 83% · MTP 73% (3.20 tok/fw) · **matmul-bound** (matmul 130 s vs disk 58 s) — unified memory, CUDA placement tier neutral; the lever here is an i8mm compute kernel, not placement |
 Takeaways: with 24 GB of RAM the engine auto-caps the expert cache to 2 slots/layer, so decode stays cold even on a disk 2–2.7× faster than the dev box — **on small-RAM machines the RAM cap, not the disk, is the binding constraint**, exactly as the table above predicts; `--topp 0.7` alone bought a clean 1.6× end-to-end speedup. The M5 Max datapoint lands right on the table's second row: **~1 tok/s of a 744B model on a laptop SSD** — and its 14 GB/s disk shifts the bottleneck back to RAM budget and kernels. The Framework 13 rows are the cache thesis proven end-to-end on one machine: 0.29 → 0.37 tok/s (hit 28% → 66%, speculation finally engaging at 52% acceptance) just by giving the cache its RAM — int8 MTP head + a bigger cap + the learned pin. The cap part is now automatic (cap auto-raise, 2026-07-10). The 9950X pair is the cleanest bottleneck experiment yet — same machine, same history, only the disk swapped: ×5.8 disk bandwidth bought ×2.9 tokens, and the profile **flipped from 66% disk to 57% matmul**. But the crossover depends on the CPU kernel: the 9800X3D row ([#101](https://github.com/JustVugg/colibri/issues/101)) shows that with the OMP hot-team tuning on, the AVX-512 CPU matmul is fast enough that even a **10 GB/s NVMe stays disk-bound** — and there the **CUDA expert tier buys ≈ 0%**, because the CPU already matches the 5090 on expert matmul. The GPU tier earns its VRAM only when the CPU is the weak link, not by default. (Honest correction from #101: an earlier version of that report ran with the OMP tuning off, which manufactured a false matmul-bound crossover and a false +14% for CUDA — neither survived a clean re-run.)
 ## Quality benchmark — help wanted
 **First measurement is in** ([#108](https://github.com/JustVugg/colibri/issues/108), thanks dnnspaul): the int4 container scored **62.5% mean acc_norm** on hellaswag/arc/mmlu (0-shot log-likelihood, n=40) — below the 85–95% published for full-precision GLM-5.2, but **the gap is not yet attributable to quantization.** Two confounds sit in the way: (1) 0-shot log-likelihood MC scoring badly underserves a *reasoning* model like GLM-5.2 (it never gets to think), so a large gap is expected even at fp16; (2) n=40 is ±14pp. The **decisive experiment** is the OLMoE fp16-vs-int4 A/B under this same harness (small enough to run both precisions) — that delta *is* the quantization cost with the scoring protocol cancelled out. Until it's run, 62.5% is a datapoint, not a verdict.
 The code is here and ready; one command runs it end to end (it auto-downloads the datasets on first use):
 ```bash
 cd c
 pip install tokenizers datasets                # in addition to the convert deps above
 ./coli bench                                   # hellaswag, arc_challenge, mmlu — 40 questions each
 ./coli bench hellaswag --limit 200             # one task, more questions
 ./coli bench mmlu arc_challenge --ram 100      # pick tasks, set a RAM budget
 ```
 It prints per-task accuracy (log-likelihood scoring, EleutherAI-harness style). **If you can run the OLMoE fp16-vs-int4 A/B (or a large-n GLM run), please open an issue with the numbers** — it's the measurement that turns 62.5% into either "int4 is fine, scoring artifact" or "quantization is the ceiling, grouped-scale is the priority."
 ## Supporting the project
-colibrì is a one-person project, written and tested entirely on a 12-core laptop with 25 GB of RAM — the numbers above are the ceiling of what I can measure at home. If this project is useful or interesting to you and you'd like to support its development (better test hardware translates *directly* into a faster engine for everyone: real NVMe scaling data, bigger pinned caches, int2/int3 quality sweeps on real benchmarks), you can:
+colibrì started as a one-person project on a 12-core laptop with 25 GB of RAM;
 today its numbers come from a community of real machines. If it's useful to you:
 - ⭐ star the repo and share it;
- 🐛 open issues with benchmark numbers from your hardware;
+- 🐛 open issues with benchmark numbers from your hardware — datapoints move
- 💬 reach out via GitHub issues if you'd like to sponsor development or donate hardware.
+  this project more than anything else;
-
+- 💬 reach out via GitHub issues to sponsor development or donate hardware.
 Every contribution, from a datapoint to a disk, moves the ceiling.
 ## Repo layout
@@ -662,20 +217,20 @@ c/
 ├── tools/                offline conversion, fixtures and benchmarks
 ├── scripts/              long-running conversion helpers
 └── tests/                dependency-free C and Python tests
-web/                      browser UI (pure OpenAI-API client, community-maintained)
+web/                      browser UI (pure OpenAI-API client)
 desktop/                  Tauri v2 desktop shell wrapping the web UI
 docs/                     reference docs, experiments, media
 ```
 The runtime path intentionally stays flat and readable: `glm.c` plus its small
-headers. Auxiliary Python and shell tooling is grouped separately and is never a
+headers. From the repository root, `make`, `make check`, and `make clean`
-runtime dependency of the engine.
+delegate to the engine Makefile.
 From the repository root, `make`, `make check`, and `make clean` delegate to the
 engine Makefile. Existing commands run from `c/` continue to work unchanged.
 ## Why "colibrì"
-The hummingbird weighs a few grams, hovers in place, and visits a thousand flowers a day. This engine keeps a 744-billion-parameter giant alive on hummingbird rations: 25 GB of RAM, twelve CPU cores, and a lot of disk patience.
+The hummingbird weighs a few grams, hovers in place, and visits a thousand
 flowers a day. This engine keeps a 744-billion-parameter giant alive on
 hummingbird rations: 25 GB of RAM, twelve CPU cores, and a lot of disk patience.
 ## License
@@ -0,0 +1,110 @@
 # OpenAI-compatible API, KV contexts & web UI
 ## `coli serve`
 `coli serve` keeps one model process loaded and exposes a text-only
 OpenAI-compatible HTTP API. The gateway uses only the Python standard library;
 inference still runs in the same dependency-free C engine.
 ```bash
 cd c
 COLI_MODEL=/nvme/glm52_i4 COLI_API_KEY=local-secret ./coli serve \
  --host 127.0.0.1 --port 8000 --model-id glm-5.2-colibri
 curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Authorization: Bearer local-secret' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "glm-5.2-colibri",
    "messages": [{"role": "user", "content": "Hello"}],
    "stream": true
  }'
 ```
 Implemented endpoints are `GET /v1/models`, `GET /v1/models/{model}`,
 `POST /v1/chat/completions`, and legacy `POST /v1/completions`. Chat and
 completion requests support JSON responses, SSE streaming, usage counts,
 `max_tokens`/`max_completion_tokens`, `temperature`, and `top_p`. The extension
 `enable_thinking: true` enables GLM-5.2's reasoning block; the standard
 `reasoning_effort` field also enables it unless set to `none`.
 The server is deliberately text-only and serves one generation at a time: the
 744B model stays in one persistent process, so concurrent HTTP requests queue
 instead of loading duplicate model copies. Tools, image/audio input, custom
 stop sequences, log probabilities, and token penalties return an explicit error
 rather than being silently ignored. The default bind address is localhost; set
 `COLI_API_KEY` before exposing the server beyond the machine.
 Browser access from the Vite development server and Tauri local origins is
 enabled by default. Repeat `--cors-origin https://your-ui.example` to allow
 another exact origin, or use `--cors-origin '*'` only on a trusted local
 network.
 The engine owns its KV contexts, so HTTP generation uses a bounded FIFO
 admission queue instead of pretending to run unsafe parallel sequences.
 Configure it with `--max-queue N` (default 8) and `--queue-timeout SECONDS`
 (default 300), or the `COLI_MAX_QUEUE` / `COLI_QUEUE_TIMEOUT` environment
 variables. Saturated and timed-out requests receive OpenAI-shaped HTTP 429
 errors before streaming headers are sent. `GET /health` exposes
 active/queued/completed/rejected counters, and successful generation responses
 include `x-colibri-queue-wait-ms`.
 ## Isolated KV contexts
 `coli serve --kv-slots N` allocates up to 16 independent sequence contexts.
 Requests select one with the optional integer `cache_slot` field; ordinary
 OpenAI clients omit it and keep the original slot 0 behavior.
 ```json
 {
  "model": "glm-5.2-colibri",
  "messages": [{"role": "user", "content": "Continue this conversation"}],
  "cache_slot": 1
 }
 ```
 Each slot owns its token history, compressed MLA/DSA KV memory, MTP window, and
 crash-safe persistence file (`.coli_kv`, `.coli_kv.1`, ...). The engine matches
 each request's tokenized prompt against the slot's history and reuses the common
 KV prefix, so stateless HTTP turns keep their cache across requests and even
 across engine restarts. Use `COLI_KV_SLOTS=N` as the environment equivalent.
 Start small: at the default 4096-token context, every slot costs hundreds of MB.
 ## Web dashboard
 One command serves the OpenAI-compatible API **and** the web console on the
 same port, then opens your browser when the engine is ready:
 ```bash
 cd web && npm install && npm run build   # once
 ./coli web --model <model-dir>
 ```
 What you get:
 - **Chat** with live metrics: a flashing token counter while generating, then
  tok/s, time-to-first-token, prompt→completion counts and queue wait;
 - **Runtime panel**: your hardware (CPU, GPUs + VRAM, RAM, cores), the
  scheduler, and the live expert-tier bar — how many of the 19,456 experts sit
  in VRAM / RAM / disk right now;
 - **Brain**: the whole model as a 76×256 cortex, one cell per expert. Colour =
  tier, brightness = routing heat, and the experts routed in each turn flash
  white and decay — you watch the model think. Hover any cell for its tier,
  heat and [measured topic affinity](https://github.com/JustVugg/colibri/issues/175);
 - **Atlas**: the measured expert atlas as a 3-D galaxy (publish `experts.json`
  from `tools/expert_atlas/analyze.py --web`).
 The dashboard talks to the engine over a small line protocol and plain JSON
 endpoints — nothing heavier than the engine itself. `web/` is a pure OpenAI-API
 client (React + TypeScript) and also works against any other compatible
 endpoint; the terminal `coli chat` remains the first-class interface.
 The layout is responsive down to phone widths, and the sidebar carries the full
 telemetry stack — hardware, scheduler, tier bar, per-turn time breakdown, tok/s
 trend and per-GPU expert counts:
 <p align="center">
  <img src="media/colibri-mobile.png" width="270" alt="the dashboard on a phone-sized viewport">
  &nbsp;&nbsp;
  <img src="media/colibri-metrics.png" width="300" alt="the telemetry sidebar">
 </p>
@@ -0,0 +1,140 @@
 # Benchmarks & measured numbers
 Everything on this page is a measurement, not a promise. If you run colibrì on
 hardware not listed here, **please open an issue with your numbers** — real
 datapoints are what move this project.
 ## Reference numbers (the original dev box: WSL2, 12 cores, 25 GB RAM, NVMe via VHDX)
 Detailed GPU experiment: [GLM-5.2 on 6× RTX 5090](experiments/glm52-6x5090-2026-07-12.md) —
 full expert residency across VRAM+RAM reaches **6.84 tok/s** single-request decode.
 | metric | value |
 |---|---|
 | model on disk (int4 container) | ~370 GB |
 | resident RAM (dense, int4) | 9.9 GB |
 | load time | ~30 s |
 | peak RSS during chat | ~20 GB (auto-capped) |
 | cold decode cost | ~11 GB disk reads/token (75 layers × 8 experts) |
 | disk ceiling (this dev box's drive) | ~1 GB/s → ~0.05–0.1 tok/s cold |
 | MTP speculation (int8 head) | 2.2–2.8 tok/forward measured ([#8](https://github.com/JustVugg/colibri/issues/8)) |
 This is not fast. It is a 744B frontier-class model **answering correctly on a
 machine that costs less than one H100 fan**. Warm cache, pinned hot experts and
 MTP push the useful-response latency down considerably; the physics of the disk
 does the rest.
 ### SSD note
 Cold starts are heavy on random reads (~11 GB/token), but reads don't
 meaningfully wear an SSD — colibrì's streaming is read-only. The real concerns
 under heavy use are (1) **swap traffic** if the system runs out of RAM (writes
 do wear the drive — keep a sane `--ram` budget; colibrì's auto-budget is designed
 to stay clear of swap) and (2) **sustained thermals**: hours at full read duty
 cycle will heat cheaper drives. Monitor drive temperature and health.
 ## Test your machine, in order
 ```bash
 cd c && ./setup.sh                 # build + architecture self-test (expects 32/32)
 # 1) measure YOUR disk the way the engine uses it (parallel 19 MB random reads):
 gcc -O2 -fopenmp iobench.c -o iobench
 ./iobench /path/to/glm52_i4/out-00069.safetensors 19 64 8 0   # buffered, 8 threads
 ./iobench /path/to/glm52_i4/out-00069.safetensors 19 64 8 1   # O_DIRECT (bypass cache)
 # Caveat (#86): iobench reads a bounded ~1 GB shard, so buffered reads on a big-RAM box
 # report the PAGE CACHE, not the disk. Use the O_DIRECT run (arg 1) for a true number, and
 # run it on a shard you haven't touched this session (a prior buffered run caches its pages).
 # On macOS there is no O_DIRECT — iobench uses F_NOCACHE, which stops *new* caching but can't
 # evict pages a prior buffered run already resident-mapped, so a macOS "O_DIRECT" figure right
 # after a buffered run still reads cache. Reboot or use a fresh shard for a real cold read.
 # 2) chat; watch the per-turn stats line (tok/s, expert hit-rate, RSS):
 COLI_MODEL=/path/to/glm52_i4 ./coli chat
 # 3) record expert usage, then pin the hottest experts in your spare RAM:
 STATS=stats.txt ./coli chat
 PIN=stats.txt PIN_GB=20 ./coli chat        # scale PIN_GB to your free RAM
 # 4) quality benchmarks (MMLU/HellaSwag/ARC):
 ./coli bench
 ```
 ## Back-of-envelope predictions
 Decode is disk-bound: a cold token costs ~11.4 GB of expert reads; MTP
 speculation roughly halves the effective cost *once the cache is warm*; RAM
 turns cold reads into free cache hits.
 | machine | expected |
 |---|---|
 | the dev box (WSL2 VHDX, ~1 GB/s, 25 GB RAM) | ~0.05–0.1 tok/s cold — proven baseline |
 | native Linux, PCIe4 NVMe (~3–5 GB/s random), 32 GB | ~0.5–1 tok/s |
 | PCIe5 NVMe or 2×NVMe RAID0 (~8–12 GB/s), 64 GB (PIN ~40 GB of hot experts) | ~2–4 tok/s |
 | 128–256 GB RAM, 12 cores (hot experts cached) | ~2–4 tok/s — matmul-bound: ~80 GFLOP/token vs ~250 GFLOP/s of our AVX2 kernels |
 | same RAM + 24–32 cores, or AVX-512/VNNI kernels | ~5–15 tok/s — interactive; kernel work is the multiplier |
 These are estimates, not measurements.
 ## Community benchmarks (measured)
 Real numbers from real machines, stock build (`setup.sh`, gcc 13), greedy decoding, `--ngen 32`, MTP active:
 | machine | disk (iobench, 19 MB × 64, 8 threads) | config | measured |
 |---|---|---|---|
 | Intel Core Ultra 7 270K Plus (24 threads) · WSL2 · 24 GB RAM · NVMe VHDX ([#2](https://github.com/JustVugg/colibri/issues/2)) | 1.96 GB/s buffered · 2.74 GB/s O_DIRECT | default | 0.07 tok/s · expert hit 3–4% · RSS 14.1 GB |
 | 〃 | 〃 | `--topp 0.7` | **0.11 tok/s** · expert hit 11% · RSS 14.7 GB |
 | Apple M5 Max (18 cores) · macOS · 128 GB unified · internal SSD ([#4](https://github.com/JustVugg/colibri/issues/4), [#5](https://github.com/JustVugg/colibri/issues/5)) | ~4 GB/s cold (the 14.2 GB/s reading was cache-influenced — see note) | default, MTP off | **1.06 tok/s** · expert hit 23% · RSS 21.8 GB |
 | Apple M5 Max · macOS · 128 GB unified · 2 TB SSD · **Metal backend** ([#72](https://github.com/JustVugg/colibri/pull/72), [#87](https://github.com/JustVugg/colibri/issues/87)) | (macOS O_DIRECT figure unreliable — see note) | Metal on · `--ram 96` · 39.7 GB warm pin · MTP off | **1.83 tok/s** · expert hit 66% · warmed 1.11 → 1.83 over the run |
 | 〃 · 46.9 GB pin (2.94M-selection history) · `--ram 110`, 1024-token run ([#103](https://github.com/JustVugg/colibri/issues/103)) | 〃 | Metal on (experts + attention) · MTP off | **2.06 tok/s** · hit 72.5% · coherent output |
 | Mac Mini M4 Pro · macOS · **48 GB** unified · **Metal backend** ([#107](https://github.com/JustVugg/colibri/issues/107)) | 6.59 GB/s F_NOCACHE (fresh shard) | Metal on · `--ram 38` | **0.30 tok/s** (vs 0.18 CPU-only) |
 | Epyc 9654 ES · Linux · 4x16GB DDR5-4800-rdimm · Samsung PCIe Gen3 x4 NVME SSD | — | `MTP=1 DIRECT=1` | 0.31 tok/s · expert hit 35% · RSS 21.52 GB |
 | Ryzen AI 9 HX 370 (Framework 13) · Arch Linux · 128 GB · WD SN850X, BTRFS zstd ([#12](https://github.com/JustVugg/colibri/issues/12)) | — | int8 MTP head · `--cap 32` · 46.7 GB auto-learned PIN | **0.37 tok/s** · expert hit 66% · MTP acceptance 52% (2.59 tok/fw) · RSS 105 GB |
 | Ryzen 9 9950X (32 threads) · Linux · 123 GB · Crucial P3 QLC Gen3 ([#31](https://github.com/JustVugg/colibri/issues/31)) | 1.51 GB/s buffered | default, 2 runs from cold | 0.10 tok/s · hit 53% · profile 66% disk |
 | 〃 same machine, model moved to a Samsung 9100 PRO PCIe 5.0 ([#31](https://github.com/JustVugg/colibri/issues/31)) | **8.81 GB/s** O_DIRECT | 〃 (usage history retained) | **0.28 tok/s** · hit 57% · profile flips: 32% disk / **57% matmul** |
 | Ryzen AI Max+ 395 (Framework Desktop) · Ubuntu · 128 GB LPDDR5x · Intel Optane 905p PCIe 3.0 ([#39](https://github.com/JustVugg/colibri/issues/39)) | 3.27 GB/s buffered | int8 MTP head · fresh history (pure LRU, auto-raised cap 65) | 0.16 tok/s · hit 57% · profile 49% disk / 47% matmul |
 | 〃 five runs later — learned pin 47.6 GB ([#39](https://github.com/JustVugg/colibri/issues/39)) | 〃 | `--temp 0.7 --topp 0.7` | **0.40 tok/s** · hit 71% |
 | Ryzen 7 9800X3D (16T) · WSL2 · 70 GB RAM · Samsung 9100 PRO PCIe 5.0 · RTX 5090 ([#101](https://github.com/JustVugg/colibri/issues/101)) | **10.51 GB/s** O_DIRECT | MTP off · learned pin 24 GB · hit 54% · OMP hot-team on | **0.41 tok/s** · disk-bound (36.5 s disk vs 24.0 s matmul) · **CUDA expert tier ≈ 0%** (AVX-512 CPU matches the 5090) · `--topp 0.7` → **0.52 tok/s** |
 | EPYC 7443 (24C/48T, Zen3 AVX2) · Linux · **430 GB RAM** · NVMe RAID-Z1 via TrueNAS VM ([#104](https://github.com/JustVugg/colibri/issues/104)) | ~1 GB/s (VM overhead) | 77.5 GB pin · cap auto-raised to 194/layer · MTP off | **1.00 tok/s** · **hit 98%** · disk eliminated → **RAM-bandwidth + matmul bound** |
 | Intel i5-12600K (10C/16T, AVX2) · **native Windows 11, no WSL** · 32 GB · MinGW GCC 16.1 ([#113](https://github.com/JustVugg/colibri/issues/113)) | buffered (no O_DIRECT on MinGW) | int8 MTP head · cold, small-RAM (cap ~2/layer) | **0.08 tok/s** · hit 3.7% · **MTP 57% acceptance** — first native-Windows datapoint |
 | Ryzen 9 9950X3D2 (16C/32T, avx512-vnni) · native Linux · 121 GB · Samsung 9100 PRO **PCIe Gen5** · RTX 5090 (28 GB expert tier, 1475 pinned) ([#120](https://github.com/JustVugg/colibri/issues/120)) | **11.48 GB/s** O_DIRECT | `MTP=0 DIRECT=1 PIPE_WORKERS=16 PREFETCH=1` | **1.23 tok/s** |
 | Ryzen AI Max+ 395 (Strix Halo, 16C/32T Zen5, avx512-vnni) · Arch Linux · 128 GB unified LPDDR5x · SK hynix P41 PCIe 4.0 ([#124](https://github.com/JustVugg/colibri/issues/124)) | — | `DIRECT=1 PIPE=1 --topp 0.7` · auto-pin | 0.06 cold → **1.10 tok/s** sustained · later **1.83 tok/s** on current dev with `DIRECT=1 PIPE=1 PILOT_REAL=1 PILOT_TWO=1` ([#200](https://github.com/JustVugg/colibri/issues/200)) |
 | Intel Core Ultra 9 185H (16C/22T, avx-vnni) · **native Windows 11, no WSL** · 32 GB · Crucial P3 QLC NTFS · RTX 5070 Ti ([#128](https://github.com/JustVugg/colibri/issues/128), [#273](https://github.com/JustVugg/colibri/issues/273)) | — | int8 MTP head · warm cache · GPU-resident pipeline at decode | 0.03 cold → 0.5 warm CPU → **1.07 tok/s** with the pipe2 decode gate (#274) |
 | Dell Pro Max GB10 (DGX Spark: Grace, **aarch64 i8mm/sve2**) · Linux · 121 GB unified LPDDR5x · GB10 sm_121 ([#136](https://github.com/JustVugg/colibri/issues/136), [#161](https://github.com/JustVugg/colibri/issues/161)) | **5.58 GB/s** O_DIRECT | int8 MTP head · warm cache | 0.50 tok/s warm · **2.4 tok/s full-k8**, **3.33 tok/s** with `CACHE_ROUTE` (#199) |
 | **6 × RTX 5090 · dual Xeon Silver 4510 · 251 GB** (author's rig, [experiment log](experiments/glm52-6x5090-2026-07-12.md)) | NVMe | `CUDA_EXPERT_GB=auto PIN_GB=all` full residency · `COLI_CUDA_PIPE=2 TC_W4A16` · DRAFT=0 | **5.8–6.8 tok/s** decode · TTFT ~13 s · hit 89–100% |
 ### Takeaways
 With 24 GB of RAM the engine auto-caps the expert cache to 2 slots/layer, so
 decode stays cold even on a fast disk — **on small-RAM machines the RAM cap, not
 the disk, is the binding constraint**; `--topp 0.7` alone bought a clean 1.6×
 end-to-end speedup. The 9950X pair is the cleanest bottleneck experiment: same
 machine, same history, only the disk swapped — ×5.8 disk bandwidth bought ×2.9
 tokens, and the profile **flipped from 66% disk to 57% matmul**. But the
 crossover depends on the CPU kernel: with OMP hot-team tuning on, an AVX-512 CPU
 can match an RTX 5090 on expert matmul ([#101](https://github.com/JustVugg/colibri/issues/101)),
 so **the GPU tier earns its VRAM only when the CPU is the weak link**. On
 multi-socket hosts, NUMA placement is a further lever: interleaving the resident
 weights across nodes measured **+13% (2-socket) and +40% (4-socket CPU-only)**
 ([#82](https://github.com/JustVugg/colibri/issues/82)) — but never blanket-interleave
 a GPU host (measured 10× regression via the DMA staging pages).
 ## Quality benchmark
 **Measured** ([#108](https://github.com/JustVugg/colibri/issues/108)): the int4
 container scored **62.5% mean acc_norm** on hellaswag/arc/mmlu (0-shot
 log-likelihood, n=40) — but 0-shot MC scoring underserves a reasoning model, and
 the OLMoE fp16-vs-int4 A/B under the same harness measured the pure quantization
 cost at **-8.2pp**, concentrated on the hardest task (per-row int4 scales erode
 the small logit margins hard questions depend on — grouped scales recover ~63%
 of that loss, see [#225](https://github.com/JustVugg/colibri/issues/225)). The
 scale-granularity/rotation/lattice ablation lives in
 `tools/quant_ablation.py` ([#81](https://github.com/JustVugg/colibri/issues/81)).
 ```bash
 cd c
 pip install tokenizers datasets
 ./coli bench                                   # hellaswag, arc_challenge, mmlu — 40 questions each
 ./coli bench hellaswag --limit 200             # one task, more questions
 ./coli bench mmlu arc_challenge --ram 100      # pick tasks, set a RAM budget
 ```
@@ -0,0 +1,104 @@
 # CUDA backend (Linux)
 colibrì includes an opt-in CUDA backend for model-resident tensors. Streaming
 experts deliberately remain on the original CPU path: copying an expert from
 NVMe to the GPU on every use would only replace the disk bottleneck with a PCIe
 bottleneck. Resident quantized tensors are uploaded lazily once and reused.
 ```bash
 cd c
 make cuda-test CUDA=1                  # q8/q4/q2/f32 kernel correctness
 make CUDA=1
 # optional dense-path experiment (hot experts are configured below)
 COLI_CUDA=1 COLI_GPU=0 CUDA_DENSE=1 SNAP=/nvme/glm52_i4 ./glm 64 4 4
 ```
 Requirements: Linux, an NVIDIA driver, and a CUDA Toolkit under
 `/usr/local/cuda` (override with `CUDA_HOME=/path/to/cuda`).
 `CUDA_ARCH=native` builds for the GPU in the current machine. Requesting CUDA
 with a CPU-only binary, an invalid device, or an unavailable runtime fails at
 startup instead of silently falling back. For Windows, see
 [windows.md](windows.md) (runtime DLL path).
 ## The VRAM expert tier
 A measured `PIN` profile promotes its hottest experts into a persistent VRAM
 tier while keeping the rest in RAM:
 ```bash
 STATS=stats.txt SNAP=/nvme/glm52_i4 ./glm 64 4 4   # collect routing frequencies first
 COLI_CUDA=1 COLI_GPU=0 CUDA_EXPERT_GB=16 \
 PIN=stats.txt PIN_GB=160 SNAP=/nvme/glm52_i4 ./glm 64 4 4
 # multi-GPU expert tier, 150 GB total budget across six 32 GB devices
 COLI_CUDA=1 COLI_GPUS=0,1,2,3,4,5 CUDA_EXPERT_GB=150 \
 CUDA_DENSE=1 PIN=stats.txt PIN_GB=300 RAM_GB=226 \
 SNAP=/nvme/glm52_i4 ./glm 64 4 4
 # large-RAM host: fill safe VRAM, then keep every remaining expert in RAM
 COLI_CUDA=1 COLI_GPUS=0,1,2,3,4,5 CUDA_EXPERT_GB=auto \
 CUDA_DENSE=1 COLI_CUDA_ATTN=1 PIN=stats.txt PIN_GB=all RAM_GB=auto \
 SNAP=/nvme/glm52_i4 ./glm 64 4 4
 ```
 Selected experts are uploaded during startup, so capacity failures occur before
 inference. The budget is clamped against free VRAM after reserving the projected
 dense resident set and 2 GB of runtime headroom per device. With `COLI_GPUS`,
 `CUDA_EXPERT_GB` is a total budget across the device set; experts are assigned
 whole to the least-loaded device that can hold them. Multi-GPU runs default to
 `PIN_FILL=1` (measured hot set first, then unused VRAM filled with zero-heat
 experts) and `CUDA_RELEASE_HOST=1` (RAM copy released after upload, reloaded
 from disk only if CUDA later fails).
 `CUDA_EXPERT_GB=auto` fills each device up to measured free memory minus
 projected dense tensors and headroom. `PIN_GB=all` then loads the remaining
 routed experts into RAM **up to the `--ram` budget** (it clamps — [#229](https://github.com/JustVugg/colibri/issues/229)),
 eliminating decode-time disk misses when capacity permits. This mode is intended
 for dedicated high-memory inference hosts.
 ### Full-residency reference result (6× RTX 5090, 251 GiB host)
 `CUDA_EXPERT_GB=auto PIN_GB=all` selected a 176.7 GB VRAM tier + 191.3 GB RAM
 tier (all 19,456 experts resident), adapting the VRAM tier every 16 tokens.
 With the GPU-resident pipeline (`COLI_CUDA_PIPE=2`) and Tensor-Core W4A16
 dispatch (`COLI_CUDA_TC_W4A16=1`), 96-token greedy decode measured
 **5.8–6.8 tok/s** (TTFT ~13 s; 1571-token prefill ~122 s then 4.2 tok/s).
 Full experiment log: [experiments/glm52-6x5090-2026-07-12.md](experiments/glm52-6x5090-2026-07-12.md).
 These are host-specific capacity results, not portable defaults.
 ## The GPU-resident pipeline (`COLI_CUDA_PIPE`)
 `COLI_CUDA_PIPE=2` keeps the residual stream on-device across layers: rmsnorms,
 residual adds, router GEMMs and the shared expert run on the GPU while the CPU
 expert loop runs uninterrupted, with batched attention and grouped expert
 uploads at prefill. On a single-GPU host this also pays at decode (S=1):
 **+49%** measured on a 5070 Ti ([#273](https://github.com/JustVugg/colibri/issues/273)/#274);
 on multi-GPU hosts the per-layer P2P hops cancel the gain, so the decode gate is
 device-count aware. `COLI_CUDA_TC_W4A16=1` enables Tensor-Core int4×fp16 mixed
 dispatch for batched rows (pays at ≥16 rows).
 ## Notes and limitations
 - Text-mode timing reports prefill separately from decode.
 - MTP speculation defaults off on CUDA (cold draft routes increase expert
  traffic); explicit `DRAFT=n` overrides. Since #294, `SPEC_PIN=1` keeps
  draft/verify kernels consistent when speculation is on.
 - Devices use independent contexts; a single expert is not sharded. Kernels are
  correctness-first custom kernels.
 - Profile quality matters more than raw VRAM capacity: the same 150 GB tier
  measured 0.94–1.64 tok/s hot-first vs 0.29 tok/s filled without routing heat.
 - The GPU tier earns its VRAM only when the CPU is the weak link — a tuned
  AVX-512 CPU can match a 5090 on expert matmul
  ([#101](https://github.com/JustVugg/colibri/issues/101)).
 ## Reproducible backend A/B without the full checkpoint
 ```bash
 cd c
 python tools/make_glm_bench_model.py --output /nvme/colibri-bench-medium --device cuda
 python tools/benchmark_cuda_fixture.py --model /nvme/colibri-bench-medium --gpu 0
 ```
 The 313M-parameter fixture has random weights and is not a language model. It
 preserves the real MLA/MoE/streaming shapes to compare CPU streaming, dense-only
 CUDA, CPU hot-store, and CUDA hot-expert execution with identical replay tokens.
@@ -0,0 +1,30 @@
 # Metal backend (Apple Silicon, experimental)
 On Apple Silicon the decode profile is matmul-bound, and unified memory removes
 the PCIe copy tax that keeps CUDA's streaming experts on the CPU — so colibrì
 has an opt-in Metal backend that runs the **routed-expert SwiGLU (batched,
 zero-copy from the RAM slabs)**, the **fused decode attention** (full MLA layer
 in one command buffer, S≤4), and **prefill's large GEMMs** on the GPU.
 Token-exact vs the CPU path.
 ```bash
 cd c
 make glm METAL=1          # macOS only; no Xcode needed (shader compiles at runtime)
 make metal-test           # standalone kernel/attention correctness vs CPU reference
 COLI_METAL=1 COLI_MODEL=/path/glm52_i4 ./coli chat --ram 96
 ```
 Measured on an M4 Max (128 GB, warm cache, MTP on): CPU 0.30 → Metal
 **0.42 tok/s (~1.4×)** (best config adds `DIRECT=1`; ~3× vs this machine's
 first cold run). An M5 Max with a 46.9 GB learned pin reached **2.06 tok/s**
 ([#103](https://github.com/JustVugg/colibri/issues/103); see also the
 [M5 Max performance report](METAL-M5MAX-PERF-REPORT.md)).
 Key design points: Metal's ~5 ms submit latency makes per-matmul dispatch a
 loss — everything is batched into few command buffers per layer, and the
 resident experts' GPU work is submitted *before* the missed experts' disk reads
 so I/O and compute overlap. `COLI_METAL_GEMM_MIN` tunes the prefill GEMM row
 threshold (default 16). Streaming, cache, MTP, DSA and the persistence formats
 are unchanged; every GPU path falls back to the CPU per-block on any fault.
 Numerics are dequant→f32-MAC (same as the CUDA tier); greedy outputs are
 byte-identical to the CPU engine.
@@ -0,0 +1,110 @@
 # Tuning & runtime knobs
 Everything here is opt-in; the defaults are chosen so a plain `./coli chat`
 is safe on any machine. See also [SETTINGS.md](SETTINGS.md) and
 [ENVIRONMENT.md](ENVIRONMENT.md) for the full variable inventory.
 ## The knobs that matter most
 | knob | what it does |
 |---|---|
 | `--temp T` | token sampling temperature (default 0.7 + nucleus 0.90 — tuned for int4; 0 = greedy) |
 | `--topp 0.7` | adaptive expert top-p (30–40% less disk; lossy — prints a warning) |
 | `--ngen N` | max tokens per answer (`:more` in chat continues a truncated one) |
 | `--repin N` | adapt RAM/VRAM hot experts every N emitted tokens |
 | `RAM_GB=<n>` | claim more RAM for the expert cache than the conservative auto-detect |
 | `PIN=stats PIN_GB=g` | pin the hottest experts from a measured usage profile |
 | `DRAFT=n` | MTP draft depth (0 disables speculation) |
 | `GRAMMAR=g.gbnf` | grammar-forced drafts for constrained JSON/NDJSON output ([docs](grammar-draft.md)) |
 | `THINK=1` | enable GLM-5.2's reasoning block |
 | `PILOT=1` | router-lookahead disk prefetch (see below) |
 | `URING=1` | Linux-only batched expert I/O (implies `PIPE=1`) |
 | `PIPE=0` | disable the async expert-load pool (default ON — overlaps `pread` with matmul, −18% disk service) |
 | `DIRECT=1` | O_DIRECT expert reads (measured **+65%** alone on a Strix Halo, [#200](https://github.com/JustVugg/colibri/issues/200)) |
 | `COLI_NUMA=1` | interleave resident weights across NUMA nodes on multi-socket hosts ([#82](https://github.com/JustVugg/colibri/issues/82)) |
 | `CACHE_ROUTE=1` | cache-aware max-rank routing (opt-in, [#199](https://github.com/JustVugg/colibri/issues/199)) |
 | `AUTOPIN=0` | disable the learning cache's auto-pin |
 | `CAP_RAISE=0` | don't auto-grow the expert cache |
 | `KVSAVE=0` | disable KV-cache persistence |
 | `TF=1` | teacher-forcing validation |
 ## Resource policy
 `coli plan` reports the planned hot (VRAM), warm (RAM), and cold backing (disk)
 tiers, the reason for each placement, and the expected bottleneck. The default
 `--policy quality` and `--policy balanced` modes preserve checkpoint quantization
 and router decisions unless `--topk` or `--topp` is passed; those explicit lossy
 overrides print a warning and proceed.
 Auto-tier plans size OpenMP from physical cores and bind workers across cores.
 Memory-bound quantized kernels can regress sharply when SMT siblings compete for
 limited memory channels; explicit `OMP_*` settings always take precedence.
 ```bash
 coli plan --model /models/glm52_i4 --policy quality
 coli run --auto-tier --policy quality "Explain MoE offloading"
 # Explicit research-only router reduction:
 coli run --policy experimental-fast --topk 4 "Benchmark prompt"
 ```
 Disk is an immutable recovery source, not a normal decode target. If the plan
 leaves cold expert bytes on disk, speed depends on cache hit rate; output quality
 does not.
 Cold expert reads can use a deferred pipeline: resident RAM/VRAM experts execute
 while missing experts are loaded in a bounded background I/O pool, then the cold
 results join before the layer completes. The pool engages only under `PIPE=1`;
 `PIPE_WORKERS=n` sets its worker count (default 8). Profiling reports both disk
 service time and the smaller foreground-visible wait time so overlap is explicit.
 `--policy balanced` enables lossless live placement (`REPIN=64`). At safe request
 boundaries, a per-layer LFRU score combines decaying session frequency with recent
 access and replaces at most four sufficiently colder pinned experts. `--policy
 quality` leaves live replacement off by default; `REPIN=0` always disables it.
 ## The learning cache
 The engine records which experts your usage actually routes to (`.coli_usage`
 next to the model, updated every turn) and at startup automatically pins the
 hottest ones in spare RAM — colibrì literally gets faster the more you use it.
 `PIN=auto` seeds the pin directly from the live usage history
 ([#301](https://github.com/JustVugg/colibri/pull/301)).
 **The expert cache auto-sizes to your RAM** (since 2026-07-10): the engine
 *raises* the LRU cap to fill your `--ram` budget instead of only lowering it.
 If you benchmarked colibrì before that date, rerun — your numbers were capped.
 **Live tier adaptation** (`--repin N`, opt-in): at safe turn boundaries, a
 decaying session heat map replaces cold pinned experts with hotter streamed
 experts. A 25% hysteresis and a four-swap limit prevent tier thrashing.
 Persistent `.coli_usage` remains the long-term signal and is not decayed.
 ## Router-lookahead prefetch (`PILOT=1`, experimental)
 GLM-5.2's expert routing is measurably predictable *ahead of time* — applying
 layer L+1's router to layer L's post-attention state recalls **71.6%** of the
 true top-8 (vs 41.3% for "same experts as last token"). `PILOT=1` issues
 next-layer expert readahead from a dedicated I/O thread while the current layer
 computes. `PILOT_REAL=1` moves the prefetched loads off the critical path
 (measured +11pp hit rate on a big-cache host), and `PILOT_TWO=1` folds the
 computed shared-expert into the prediction (+3% recall,
 [#200](https://github.com/JustVugg/colibri/issues/200)). On disk-saturated
 hosts hint-only PILOT can be net negative — measure on yours.
 ## Speculation and reproducibility
 Speculative decoding requires that the draft and verify paths compute the same
 function — `SPEC_PIN=1` (default since [#294](https://github.com/JustVugg/colibri/pull/294))
 pins every forward issued while drafts are live to the platform's S=1 kernel
 family. For byte-exact reproducibility across runs: `DRAFT=0`, plus `IDOT=0
 COLI_CUDA=0` if you also want kernel-family/GPU independence. Acceptance
 percentages are not comparable across engine versions under `--topp`
 ([#163](https://github.com/JustVugg/colibri/issues/163) has the full story).
 ## Conversations reopen warm
 `coli chat` persists the compressed MLA KV-cache to disk after every turn
 (`.coli_kv`, ~182 KB/token, appended incrementally, crash-safe). Close the chat,
 reopen it tomorrow — the model still remembers the whole conversation and **zero
 re-prefill happens**: validated byte-identical to an uninterrupted session.
 `:reset` clears it, `KVSAVE=0` disables it.
@@ -0,0 +1,90 @@
 # Windows 11 (native, no WSL)
 colibrì builds and runs natively on Windows 11 x86-64 with MinGW-w64. The port
 adds a `_WIN32` compatibility layer in `c/compat.h` that maps POSIX I/O to the
 Windows API (pread → ReadFile+OVERLAPPED, posix_fadvise no-op, aligned
 allocation, MoveFileEx rename, GlobalMemoryStatusEx RAM detection). All platform
 differences stay in `compat.h`; the engine source is unchanged.
 **Toolchain:** GCC via [winlibs](https://winlibs.com/) or MSYS2 MinGW-w64.
 Tested with GCC 16.1.0 (x86_64-ucrt-posix-seh).
 ```powershell
 # One-time toolchain install (pick one):
 scoop install mingw-winlibs                    # portable, no shell needed
 # or: pacman -S mingw-w64-x86_64-gcc make     # via MSYS2
 # Build (from c/ directory):
 make glm.exe            # GLM-5.2 engine (static, no DLL dependencies)
 make olmoe.exe          # OLMoE engine (same shims)
 make iobench.exe        # disk I/O benchmark
 make test-c             # run C tests
 make test-python        # run Python tests (requires python)
 # AVX-VNNI: Intel Alder Lake+ (and Meteor Lake+) CPUs have a 128-bit int8
 # dot-product instruction (VPDPBUSD) the engine can use for ~1.3x faster
 # quantized matmul. The x86-64-v3 default (portable AVX2) compiles it out;
 # build for THIS machine to enable it:
 make glm.exe ARCH=native                       # banner prints "idot: avx-vnni"
 # Verify (tiny model, 2.4 MB):
 pip install torch transformers safetensors huggingface_hub
 python tools/make_glm_oracle.py                # generate tiny oracle
 SNAP=./glm_tiny TF=1 ./glm.exe 64 16 16        # expect "32/32 positions"
 # Run with real model:
 SNAP=D:\glm52_i4 ./glm.exe 64 4 16            # batch inference
 python coli chat --model D:\glm52_i4            # interactive chat
 python coli serve --model D:\glm52_i4            # OpenAI-compatible API
 ```
 > Windows Store's `python` alias stub is the single most common native-Windows
 > trap: install real Python (python.org or `winget install Python.Python.3.12`)
 > or disable the alias under *Settings → Apps → App execution aliases*.
 ## Warmup (overnight cache priming)
 The engine's expert cache learns from your workload. The included `warmup.ps1`
 script runs `coli run` in a loop with diverse prompts to build the
 `.coli_usage` histogram unattended, so the next real session starts with a
 large, accurate hot-expert pin. Each run saves usage atomically on clean
 completion.
 ```powershell
 .\warmup.ps1 -Rounds 1 -Ngen 32               # ~60-90 min, durable progress
 ```
 ## NVIDIA GPU (optional, via runtime DLL)
 On Windows the engine is built with MinGW gcc but CUDA kernels require MSVC +
 nvcc. The split is clean: build the CUDA backend into a standalone
 `coli_cuda.dll` (nvcc + MSVC), then the host `glm.exe` loads it at runtime via
 `LoadLibrary` (`c/backend_loader.c`). The host never links cudart directly; if
 the DLL is absent the engine falls back to CPU without error.
 ```powershell
 # Prerequisites: CUDA Toolkit + MSVC Build Tools (cl.exe) + nvcc on PATH.
 # Build the DLL from a shell with the MSVC environment set (vcvars64.bat or
 # "x64 Native Tools Command Prompt for VS"):
 make cuda-dll CUDA_HOME="C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.8" CUDA_ARCH=sm_120
 # Build the host with the runtime loader (CUDA_DLL=1 adds -DCOLI_CUDA and
 # links backend_loader.o instead of cudart):
 make glm.exe CUDA_DLL=1 ARCH=native
 # Run with the GPU expert tier (8 GB VRAM budget here; scale to your free VRAM):
 $env:COLI_CUDA="1"; $env:COLI_GPU="0"; $env:CUDA_EXPERT_GB="8"
 python coli chat --model D:\glm52_i4 --topp 0.7
 ```
 The DLL exports the full `extern "C"` surface (including the #111 pipeline ABI);
 `backend_loader.c` resolves symbols via `GetProcAddress` on first use.
 `ColiCudaTensor*` is opaque to the host (stored, never dereferenced), so the
 MSVC-allocated struct is safe across the ABI boundary. `CUDA_ARCH` must match
 your GPU's compute capability (e.g. `sm_120` for Blackwell / RTX 50-series,
 `sm_89` for Ada / RTX 40-series). A one-shot `build_cuda.bat` wrapper is also
 available.
 **Measured on a single RTX 5070 Ti + Core Ultra 9 (32 GB RAM):** CPU-only 0.63
 → CUDA attention+dense 0.72 → **1.07 tok/s** with the GPU-resident pipeline at
 decode ([#273](https://github.com/JustVugg/colibri/issues/273), merged in #274).