Merge branch 't321' into trial321b

# Conflicts:
#	README.md

docs/api.md

@@ -0,0 +1,110 @@
+# OpenAI-compatible API, KV contexts & web UI
+
+## `coli serve`
+
+`coli serve` keeps one model process loaded and exposes a text-only
+OpenAI-compatible HTTP API. The gateway uses only the Python standard library;
+inference still runs in the same dependency-free C engine.
+
+```bash
+cd c
+COLI_MODEL=/nvme/glm52_i4 COLI_API_KEY=local-secret ./coli serve \
+  --host 127.0.0.1 --port 8000 --model-id glm-5.2-colibri
+
+curl http://127.0.0.1:8000/v1/chat/completions \
+  -H 'Authorization: Bearer local-secret' \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "model": "glm-5.2-colibri",
+    "messages": [{"role": "user", "content": "Hello"}],
+    "stream": true
+  }'
+```
+
+Implemented endpoints are `GET /v1/models`, `GET /v1/models/{model}`,
+`POST /v1/chat/completions`, and legacy `POST /v1/completions`. Chat and
+completion requests support JSON responses, SSE streaming, usage counts,
+`max_tokens`/`max_completion_tokens`, `temperature`, and `top_p`. The extension
+`enable_thinking: true` enables GLM-5.2's reasoning block; the standard
+`reasoning_effort` field also enables it unless set to `none`.
+
+The server is deliberately text-only and serves one generation at a time: the
+744B model stays in one persistent process, so concurrent HTTP requests queue
+instead of loading duplicate model copies. Tools, image/audio input, custom
+stop sequences, log probabilities, and token penalties return an explicit error
+rather than being silently ignored. The default bind address is localhost; set
+`COLI_API_KEY` before exposing the server beyond the machine.
+
+Browser access from the Vite development server and Tauri local origins is
+enabled by default. Repeat `--cors-origin https://your-ui.example` to allow
+another exact origin, or use `--cors-origin '*'` only on a trusted local
+network.
+
+The engine owns its KV contexts, so HTTP generation uses a bounded FIFO
+admission queue instead of pretending to run unsafe parallel sequences.
+Configure it with `--max-queue N` (default 8) and `--queue-timeout SECONDS`
+(default 300), or the `COLI_MAX_QUEUE` / `COLI_QUEUE_TIMEOUT` environment
+variables. Saturated and timed-out requests receive OpenAI-shaped HTTP 429
+errors before streaming headers are sent. `GET /health` exposes
+active/queued/completed/rejected counters, and successful generation responses
+include `x-colibri-queue-wait-ms`.
+
+## Isolated KV contexts
+
+`coli serve --kv-slots N` allocates up to 16 independent sequence contexts.
+Requests select one with the optional integer `cache_slot` field; ordinary
+OpenAI clients omit it and keep the original slot 0 behavior.
+
+```json
+{
+  "model": "glm-5.2-colibri",
+  "messages": [{"role": "user", "content": "Continue this conversation"}],
+  "cache_slot": 1
+}
+```
+
+Each slot owns its token history, compressed MLA/DSA KV memory, MTP window, and
+crash-safe persistence file (`.coli_kv`, `.coli_kv.1`, ...). The engine matches
+each request's tokenized prompt against the slot's history and reuses the common
+KV prefix, so stateless HTTP turns keep their cache across requests and even
+across engine restarts. Use `COLI_KV_SLOTS=N` as the environment equivalent.
+Start small: at the default 4096-token context, every slot costs hundreds of MB.
+
+## Web dashboard
+
+One command serves the OpenAI-compatible API **and** the web console on the
+same port, then opens your browser when the engine is ready:
+
+```bash
+cd web && npm install && npm run build   # once
+./coli web --model <model-dir>
+```
+
+What you get:
+
+- **Chat** with live metrics: a flashing token counter while generating, then
+  tok/s, time-to-first-token, prompt→completion counts and queue wait;
+- **Runtime panel**: your hardware (CPU, GPUs + VRAM, RAM, cores), the
+  scheduler, and the live expert-tier bar — how many of the 19,456 experts sit
+  in VRAM / RAM / disk right now;
+- **Brain**: the whole model as a 76×256 cortex, one cell per expert. Colour =
+  tier, brightness = routing heat, and the experts routed in each turn flash
+  white and decay — you watch the model think. Hover any cell for its tier,
+  heat and [measured topic affinity](https://github.com/JustVugg/colibri/issues/175);
+- **Atlas**: the measured expert atlas as a 3-D galaxy (publish `experts.json`
+  from `tools/expert_atlas/analyze.py --web`).
+
+The dashboard talks to the engine over a small line protocol and plain JSON
+endpoints — nothing heavier than the engine itself. `web/` is a pure OpenAI-API
+client (React + TypeScript) and also works against any other compatible
+endpoint; the terminal `coli chat` remains the first-class interface.
+
+The layout is responsive down to phone widths, and the sidebar carries the full
+telemetry stack — hardware, scheduler, tier bar, per-turn time breakdown, tok/s
+trend and per-GPU expert counts:
+
+<p align="center">
+  <img src="media/colibri-mobile.png" width="270" alt="the dashboard on a phone-sized viewport">
+  &nbsp;&nbsp;
+  <img src="media/colibri-metrics.png" width="300" alt="the telemetry sidebar">
+</p>

docs/benchmarks.md

@@ -0,0 +1,140 @@
+# Benchmarks & measured numbers
+
+Everything on this page is a measurement, not a promise. If you run colibrì on
+hardware not listed here, **please open an issue with your numbers** — real
+datapoints are what move this project.
+
+## Reference numbers (the original dev box: WSL2, 12 cores, 25 GB RAM, NVMe via VHDX)
+
+Detailed GPU experiment: [GLM-5.2 on 6× RTX 5090](experiments/glm52-6x5090-2026-07-12.md) —
+full expert residency across VRAM+RAM reaches **6.84 tok/s** single-request decode.
+
+| metric | value |
+|---|---|
+| model on disk (int4 container) | ~370 GB |
+| resident RAM (dense, int4) | 9.9 GB |
+| load time | ~30 s |
+| peak RSS during chat | ~20 GB (auto-capped) |
+| cold decode cost | ~11 GB disk reads/token (75 layers × 8 experts) |
+| disk ceiling (this dev box's drive) | ~1 GB/s → ~0.05–0.1 tok/s cold |
+| MTP speculation (int8 head) | 2.2–2.8 tok/forward measured ([#8](https://github.com/JustVugg/colibri/issues/8)) |
+
+This is not fast. It is a 744B frontier-class model **answering correctly on a
+machine that costs less than one H100 fan**. Warm cache, pinned hot experts and
+MTP push the useful-response latency down considerably; the physics of the disk
+does the rest.
+
+### SSD note
+
+Cold starts are heavy on random reads (~11 GB/token), but reads don't
+meaningfully wear an SSD — colibrì's streaming is read-only. The real concerns
+under heavy use are (1) **swap traffic** if the system runs out of RAM (writes
+do wear the drive — keep a sane `--ram` budget; colibrì's auto-budget is designed
+to stay clear of swap) and (2) **sustained thermals**: hours at full read duty
+cycle will heat cheaper drives. Monitor drive temperature and health.
+
+## Test your machine, in order
+
+```bash
+cd c && ./setup.sh                 # build + architecture self-test (expects 32/32)
+
+# 1) measure YOUR disk the way the engine uses it (parallel 19 MB random reads):
+gcc -O2 -fopenmp iobench.c -o iobench
+./iobench /path/to/glm52_i4/out-00069.safetensors 19 64 8 0   # buffered, 8 threads
+./iobench /path/to/glm52_i4/out-00069.safetensors 19 64 8 1   # O_DIRECT (bypass cache)
+# Caveat (#86): iobench reads a bounded ~1 GB shard, so buffered reads on a big-RAM box
+# report the PAGE CACHE, not the disk. Use the O_DIRECT run (arg 1) for a true number, and
+# run it on a shard you haven't touched this session (a prior buffered run caches its pages).
+# On macOS there is no O_DIRECT — iobench uses F_NOCACHE, which stops *new* caching but can't
+# evict pages a prior buffered run already resident-mapped, so a macOS "O_DIRECT" figure right
+# after a buffered run still reads cache. Reboot or use a fresh shard for a real cold read.
+
+# 2) chat; watch the per-turn stats line (tok/s, expert hit-rate, RSS):
+COLI_MODEL=/path/to/glm52_i4 ./coli chat
+
+# 3) record expert usage, then pin the hottest experts in your spare RAM:
+STATS=stats.txt ./coli chat
+PIN=stats.txt PIN_GB=20 ./coli chat        # scale PIN_GB to your free RAM
+
+# 4) quality benchmarks (MMLU/HellaSwag/ARC):
+./coli bench
+```
+
+## Back-of-envelope predictions
+
+Decode is disk-bound: a cold token costs ~11.4 GB of expert reads; MTP
+speculation roughly halves the effective cost *once the cache is warm*; RAM
+turns cold reads into free cache hits.
+
+| machine | expected |
+|---|---|
+| the dev box (WSL2 VHDX, ~1 GB/s, 25 GB RAM) | ~0.05–0.1 tok/s cold — proven baseline |
+| native Linux, PCIe4 NVMe (~3–5 GB/s random), 32 GB | ~0.5–1 tok/s |
+| PCIe5 NVMe or 2×NVMe RAID0 (~8–12 GB/s), 64 GB (PIN ~40 GB of hot experts) | ~2–4 tok/s |
+| 128–256 GB RAM, 12 cores (hot experts cached) | ~2–4 tok/s — matmul-bound: ~80 GFLOP/token vs ~250 GFLOP/s of our AVX2 kernels |
+| same RAM + 24–32 cores, or AVX-512/VNNI kernels | ~5–15 tok/s — interactive; kernel work is the multiplier |
+
+These are estimates, not measurements.
+
+## Community benchmarks (measured)
+
+Real numbers from real machines, stock build (`setup.sh`, gcc 13), greedy decoding, `--ngen 32`, MTP active:
+
+| machine | disk (iobench, 19 MB × 64, 8 threads) | config | measured |
+|---|---|---|---|
+| Intel Core Ultra 7 270K Plus (24 threads) · WSL2 · 24 GB RAM · NVMe VHDX ([#2](https://github.com/JustVugg/colibri/issues/2)) | 1.96 GB/s buffered · 2.74 GB/s O_DIRECT | default | 0.07 tok/s · expert hit 3–4% · RSS 14.1 GB |
+| 〃 | 〃 | `--topp 0.7` | **0.11 tok/s** · expert hit 11% · RSS 14.7 GB |
+| Apple M5 Max (18 cores) · macOS · 128 GB unified · internal SSD ([#4](https://github.com/JustVugg/colibri/issues/4), [#5](https://github.com/JustVugg/colibri/issues/5)) | ~4 GB/s cold (the 14.2 GB/s reading was cache-influenced — see note) | default, MTP off | **1.06 tok/s** · expert hit 23% · RSS 21.8 GB |
+| Apple M5 Max · macOS · 128 GB unified · 2 TB SSD · **Metal backend** ([#72](https://github.com/JustVugg/colibri/pull/72), [#87](https://github.com/JustVugg/colibri/issues/87)) | (macOS O_DIRECT figure unreliable — see note) | Metal on · `--ram 96` · 39.7 GB warm pin · MTP off | **1.83 tok/s** · expert hit 66% · warmed 1.11 → 1.83 over the run |
+| 〃 · 46.9 GB pin (2.94M-selection history) · `--ram 110`, 1024-token run ([#103](https://github.com/JustVugg/colibri/issues/103)) | 〃 | Metal on (experts + attention) · MTP off | **2.06 tok/s** · hit 72.5% · coherent output |
+| Mac Mini M4 Pro · macOS · **48 GB** unified · **Metal backend** ([#107](https://github.com/JustVugg/colibri/issues/107)) | 6.59 GB/s F_NOCACHE (fresh shard) | Metal on · `--ram 38` | **0.30 tok/s** (vs 0.18 CPU-only) |
+| Epyc 9654 ES · Linux · 4x16GB DDR5-4800-rdimm · Samsung PCIe Gen3 x4 NVME SSD | — | `MTP=1 DIRECT=1` | 0.31 tok/s · expert hit 35% · RSS 21.52 GB |
+| Ryzen AI 9 HX 370 (Framework 13) · Arch Linux · 128 GB · WD SN850X, BTRFS zstd ([#12](https://github.com/JustVugg/colibri/issues/12)) | — | int8 MTP head · `--cap 32` · 46.7 GB auto-learned PIN | **0.37 tok/s** · expert hit 66% · MTP acceptance 52% (2.59 tok/fw) · RSS 105 GB |
+| Ryzen 9 9950X (32 threads) · Linux · 123 GB · Crucial P3 QLC Gen3 ([#31](https://github.com/JustVugg/colibri/issues/31)) | 1.51 GB/s buffered | default, 2 runs from cold | 0.10 tok/s · hit 53% · profile 66% disk |
+| 〃 same machine, model moved to a Samsung 9100 PRO PCIe 5.0 ([#31](https://github.com/JustVugg/colibri/issues/31)) | **8.81 GB/s** O_DIRECT | 〃 (usage history retained) | **0.28 tok/s** · hit 57% · profile flips: 32% disk / **57% matmul** |
+| Ryzen AI Max+ 395 (Framework Desktop) · Ubuntu · 128 GB LPDDR5x · Intel Optane 905p PCIe 3.0 ([#39](https://github.com/JustVugg/colibri/issues/39)) | 3.27 GB/s buffered | int8 MTP head · fresh history (pure LRU, auto-raised cap 65) | 0.16 tok/s · hit 57% · profile 49% disk / 47% matmul |
+| 〃 five runs later — learned pin 47.6 GB ([#39](https://github.com/JustVugg/colibri/issues/39)) | 〃 | `--temp 0.7 --topp 0.7` | **0.40 tok/s** · hit 71% |
+| Ryzen 7 9800X3D (16T) · WSL2 · 70 GB RAM · Samsung 9100 PRO PCIe 5.0 · RTX 5090 ([#101](https://github.com/JustVugg/colibri/issues/101)) | **10.51 GB/s** O_DIRECT | MTP off · learned pin 24 GB · hit 54% · OMP hot-team on | **0.41 tok/s** · disk-bound (36.5 s disk vs 24.0 s matmul) · **CUDA expert tier ≈ 0%** (AVX-512 CPU matches the 5090) · `--topp 0.7` → **0.52 tok/s** |
+| EPYC 7443 (24C/48T, Zen3 AVX2) · Linux · **430 GB RAM** · NVMe RAID-Z1 via TrueNAS VM ([#104](https://github.com/JustVugg/colibri/issues/104)) | ~1 GB/s (VM overhead) | 77.5 GB pin · cap auto-raised to 194/layer · MTP off | **1.00 tok/s** · **hit 98%** · disk eliminated → **RAM-bandwidth + matmul bound** |
+| Intel i5-12600K (10C/16T, AVX2) · **native Windows 11, no WSL** · 32 GB · MinGW GCC 16.1 ([#113](https://github.com/JustVugg/colibri/issues/113)) | buffered (no O_DIRECT on MinGW) | int8 MTP head · cold, small-RAM (cap ~2/layer) | **0.08 tok/s** · hit 3.7% · **MTP 57% acceptance** — first native-Windows datapoint |
+| Ryzen 9 9950X3D2 (16C/32T, avx512-vnni) · native Linux · 121 GB · Samsung 9100 PRO **PCIe Gen5** · RTX 5090 (28 GB expert tier, 1475 pinned) ([#120](https://github.com/JustVugg/colibri/issues/120)) | **11.48 GB/s** O_DIRECT | `MTP=0 DIRECT=1 PIPE_WORKERS=16 PREFETCH=1` | **1.23 tok/s** |
+| Ryzen AI Max+ 395 (Strix Halo, 16C/32T Zen5, avx512-vnni) · Arch Linux · 128 GB unified LPDDR5x · SK hynix P41 PCIe 4.0 ([#124](https://github.com/JustVugg/colibri/issues/124)) | — | `DIRECT=1 PIPE=1 --topp 0.7` · auto-pin | 0.06 cold → **1.10 tok/s** sustained · later **1.83 tok/s** on current dev with `DIRECT=1 PIPE=1 PILOT_REAL=1 PILOT_TWO=1` ([#200](https://github.com/JustVugg/colibri/issues/200)) |
+| Intel Core Ultra 9 185H (16C/22T, avx-vnni) · **native Windows 11, no WSL** · 32 GB · Crucial P3 QLC NTFS · RTX 5070 Ti ([#128](https://github.com/JustVugg/colibri/issues/128), [#273](https://github.com/JustVugg/colibri/issues/273)) | — | int8 MTP head · warm cache · GPU-resident pipeline at decode | 0.03 cold → 0.5 warm CPU → **1.07 tok/s** with the pipe2 decode gate (#274) |
+| Dell Pro Max GB10 (DGX Spark: Grace, **aarch64 i8mm/sve2**) · Linux · 121 GB unified LPDDR5x · GB10 sm_121 ([#136](https://github.com/JustVugg/colibri/issues/136), [#161](https://github.com/JustVugg/colibri/issues/161)) | **5.58 GB/s** O_DIRECT | int8 MTP head · warm cache | 0.50 tok/s warm · **2.4 tok/s full-k8**, **3.33 tok/s** with `CACHE_ROUTE` (#199) |
+| **6 × RTX 5090 · dual Xeon Silver 4510 · 251 GB** (author's rig, [experiment log](experiments/glm52-6x5090-2026-07-12.md)) | NVMe | `CUDA_EXPERT_GB=auto PIN_GB=all` full residency · `COLI_CUDA_PIPE=2 TC_W4A16` · DRAFT=0 | **5.8–6.8 tok/s** decode · TTFT ~13 s · hit 89–100% |
+
+### Takeaways
+
+With 24 GB of RAM the engine auto-caps the expert cache to 2 slots/layer, so
+decode stays cold even on a fast disk — **on small-RAM machines the RAM cap, not
+the disk, is the binding constraint**; `--topp 0.7` alone bought a clean 1.6×
+end-to-end speedup. The 9950X pair is the cleanest bottleneck experiment: same
+machine, same history, only the disk swapped — ×5.8 disk bandwidth bought ×2.9
+tokens, and the profile **flipped from 66% disk to 57% matmul**. But the
+crossover depends on the CPU kernel: with OMP hot-team tuning on, an AVX-512 CPU
+can match an RTX 5090 on expert matmul ([#101](https://github.com/JustVugg/colibri/issues/101)),
+so **the GPU tier earns its VRAM only when the CPU is the weak link**. On
+multi-socket hosts, NUMA placement is a further lever: interleaving the resident
+weights across nodes measured **+13% (2-socket) and +40% (4-socket CPU-only)**
+([#82](https://github.com/JustVugg/colibri/issues/82)) — but never blanket-interleave
+a GPU host (measured 10× regression via the DMA staging pages).
+
+## Quality benchmark
+
+**Measured** ([#108](https://github.com/JustVugg/colibri/issues/108)): the int4
+container scored **62.5% mean acc_norm** on hellaswag/arc/mmlu (0-shot
+log-likelihood, n=40) — but 0-shot MC scoring underserves a reasoning model, and
+the OLMoE fp16-vs-int4 A/B under the same harness measured the pure quantization
+cost at **-8.2pp**, concentrated on the hardest task (per-row int4 scales erode
+the small logit margins hard questions depend on — grouped scales recover ~63%
+of that loss, see [#225](https://github.com/JustVugg/colibri/issues/225)). The
+scale-granularity/rotation/lattice ablation lives in
+`tools/quant_ablation.py` ([#81](https://github.com/JustVugg/colibri/issues/81)).
+
+```bash
+cd c
+pip install tokenizers datasets
+./coli bench                                   # hellaswag, arc_challenge, mmlu — 40 questions each
+./coli bench hellaswag --limit 200             # one task, more questions
+./coli bench mmlu arc_challenge --ram 100      # pick tasks, set a RAM budget
+```

docs/cuda.md

@@ -0,0 +1,104 @@
+# CUDA backend (Linux)
+
+colibrì includes an opt-in CUDA backend for model-resident tensors. Streaming
+experts deliberately remain on the original CPU path: copying an expert from
+NVMe to the GPU on every use would only replace the disk bottleneck with a PCIe
+bottleneck. Resident quantized tensors are uploaded lazily once and reused.
+
+```bash
+cd c
+make cuda-test CUDA=1                  # q8/q4/q2/f32 kernel correctness
+make CUDA=1
+# optional dense-path experiment (hot experts are configured below)
+COLI_CUDA=1 COLI_GPU=0 CUDA_DENSE=1 SNAP=/nvme/glm52_i4 ./glm 64 4 4
+```
+
+Requirements: Linux, an NVIDIA driver, and a CUDA Toolkit under
+`/usr/local/cuda` (override with `CUDA_HOME=/path/to/cuda`).
+`CUDA_ARCH=native` builds for the GPU in the current machine. Requesting CUDA
+with a CPU-only binary, an invalid device, or an unavailable runtime fails at
+startup instead of silently falling back. For Windows, see
+[windows.md](windows.md) (runtime DLL path).
+
+## The VRAM expert tier
+
+A measured `PIN` profile promotes its hottest experts into a persistent VRAM
+tier while keeping the rest in RAM:
+
+```bash
+STATS=stats.txt SNAP=/nvme/glm52_i4 ./glm 64 4 4   # collect routing frequencies first
+COLI_CUDA=1 COLI_GPU=0 CUDA_EXPERT_GB=16 \
+PIN=stats.txt PIN_GB=160 SNAP=/nvme/glm52_i4 ./glm 64 4 4
+
+# multi-GPU expert tier, 150 GB total budget across six 32 GB devices
+COLI_CUDA=1 COLI_GPUS=0,1,2,3,4,5 CUDA_EXPERT_GB=150 \
+CUDA_DENSE=1 PIN=stats.txt PIN_GB=300 RAM_GB=226 \
+SNAP=/nvme/glm52_i4 ./glm 64 4 4
+
+# large-RAM host: fill safe VRAM, then keep every remaining expert in RAM
+COLI_CUDA=1 COLI_GPUS=0,1,2,3,4,5 CUDA_EXPERT_GB=auto \
+CUDA_DENSE=1 COLI_CUDA_ATTN=1 PIN=stats.txt PIN_GB=all RAM_GB=auto \
+SNAP=/nvme/glm52_i4 ./glm 64 4 4
+```
+
+Selected experts are uploaded during startup, so capacity failures occur before
+inference. The budget is clamped against free VRAM after reserving the projected
+dense resident set and 2 GB of runtime headroom per device. With `COLI_GPUS`,
+`CUDA_EXPERT_GB` is a total budget across the device set; experts are assigned
+whole to the least-loaded device that can hold them. Multi-GPU runs default to
+`PIN_FILL=1` (measured hot set first, then unused VRAM filled with zero-heat
+experts) and `CUDA_RELEASE_HOST=1` (RAM copy released after upload, reloaded
+from disk only if CUDA later fails).
+
+`CUDA_EXPERT_GB=auto` fills each device up to measured free memory minus
+projected dense tensors and headroom. `PIN_GB=all` then loads the remaining
+routed experts into RAM **up to the `--ram` budget** (it clamps — [#229](https://github.com/JustVugg/colibri/issues/229)),
+eliminating decode-time disk misses when capacity permits. This mode is intended
+for dedicated high-memory inference hosts.
+
+### Full-residency reference result (6× RTX 5090, 251 GiB host)
+
+`CUDA_EXPERT_GB=auto PIN_GB=all` selected a 176.7 GB VRAM tier + 191.3 GB RAM
+tier (all 19,456 experts resident), adapting the VRAM tier every 16 tokens.
+With the GPU-resident pipeline (`COLI_CUDA_PIPE=2`) and Tensor-Core W4A16
+dispatch (`COLI_CUDA_TC_W4A16=1`), 96-token greedy decode measured
+**5.8–6.8 tok/s** (TTFT ~13 s; 1571-token prefill ~122 s then 4.2 tok/s).
+Full experiment log: [experiments/glm52-6x5090-2026-07-12.md](experiments/glm52-6x5090-2026-07-12.md).
+These are host-specific capacity results, not portable defaults.
+
+## The GPU-resident pipeline (`COLI_CUDA_PIPE`)
+
+`COLI_CUDA_PIPE=2` keeps the residual stream on-device across layers: rmsnorms,
+residual adds, router GEMMs and the shared expert run on the GPU while the CPU
+expert loop runs uninterrupted, with batched attention and grouped expert
+uploads at prefill. On a single-GPU host this also pays at decode (S=1):
+**+49%** measured on a 5070 Ti ([#273](https://github.com/JustVugg/colibri/issues/273)/#274);
+on multi-GPU hosts the per-layer P2P hops cancel the gain, so the decode gate is
+device-count aware. `COLI_CUDA_TC_W4A16=1` enables Tensor-Core int4×fp16 mixed
+dispatch for batched rows (pays at ≥16 rows).
+
+## Notes and limitations
+
+- Text-mode timing reports prefill separately from decode.
+- MTP speculation defaults off on CUDA (cold draft routes increase expert
+  traffic); explicit `DRAFT=n` overrides. Since #294, `SPEC_PIN=1` keeps
+  draft/verify kernels consistent when speculation is on.
+- Devices use independent contexts; a single expert is not sharded. Kernels are
+  correctness-first custom kernels.
+- Profile quality matters more than raw VRAM capacity: the same 150 GB tier
+  measured 0.94–1.64 tok/s hot-first vs 0.29 tok/s filled without routing heat.
+- The GPU tier earns its VRAM only when the CPU is the weak link — a tuned
+  AVX-512 CPU can match a 5090 on expert matmul
+  ([#101](https://github.com/JustVugg/colibri/issues/101)).
+
+## Reproducible backend A/B without the full checkpoint
+
+```bash
+cd c
+python tools/make_glm_bench_model.py --output /nvme/colibri-bench-medium --device cuda
+python tools/benchmark_cuda_fixture.py --model /nvme/colibri-bench-medium --gpu 0
+```
+
+The 313M-parameter fixture has random weights and is not a language model. It
+preserves the real MLA/MoE/streaming shapes to compare CPU streaming, dense-only
+CUDA, CPU hot-store, and CUDA hot-expert execution with identical replay tokens.

docs/metal.md

@@ -0,0 +1,30 @@
+# Metal backend (Apple Silicon, experimental)
+
+On Apple Silicon the decode profile is matmul-bound, and unified memory removes
+the PCIe copy tax that keeps CUDA's streaming experts on the CPU — so colibrì
+has an opt-in Metal backend that runs the **routed-expert SwiGLU (batched,
+zero-copy from the RAM slabs)**, the **fused decode attention** (full MLA layer
+in one command buffer, S≤4), and **prefill's large GEMMs** on the GPU.
+Token-exact vs the CPU path.
+
+```bash
+cd c
+make glm METAL=1          # macOS only; no Xcode needed (shader compiles at runtime)
+make metal-test           # standalone kernel/attention correctness vs CPU reference
+COLI_METAL=1 COLI_MODEL=/path/glm52_i4 ./coli chat --ram 96
+```
+
+Measured on an M4 Max (128 GB, warm cache, MTP on): CPU 0.30 → Metal
+**0.42 tok/s (~1.4×)** (best config adds `DIRECT=1`; ~3× vs this machine's
+first cold run). An M5 Max with a 46.9 GB learned pin reached **2.06 tok/s**
+([#103](https://github.com/JustVugg/colibri/issues/103); see also the
+[M5 Max performance report](METAL-M5MAX-PERF-REPORT.md)).
+
+Key design points: Metal's ~5 ms submit latency makes per-matmul dispatch a
+loss — everything is batched into few command buffers per layer, and the
+resident experts' GPU work is submitted *before* the missed experts' disk reads
+so I/O and compute overlap. `COLI_METAL_GEMM_MIN` tunes the prefill GEMM row
+threshold (default 16). Streaming, cache, MTP, DSA and the persistence formats
+are unchanged; every GPU path falls back to the CPU per-block on any fault.
+Numerics are dequant→f32-MAC (same as the CUDA tier); greedy outputs are
+byte-identical to the CPU engine.

docs/tuning.md

@@ -0,0 +1,110 @@
+# Tuning & runtime knobs
+
+Everything here is opt-in; the defaults are chosen so a plain `./coli chat`
+is safe on any machine. See also [SETTINGS.md](SETTINGS.md) and
+[ENVIRONMENT.md](ENVIRONMENT.md) for the full variable inventory.
+
+## The knobs that matter most
+
+| knob | what it does |
+|---|---|
+| `--temp T` | token sampling temperature (default 0.7 + nucleus 0.90 — tuned for int4; 0 = greedy) |
+| `--topp 0.7` | adaptive expert top-p (30–40% less disk; lossy — prints a warning) |
+| `--ngen N` | max tokens per answer (`:more` in chat continues a truncated one) |
+| `--repin N` | adapt RAM/VRAM hot experts every N emitted tokens |
+| `RAM_GB=<n>` | claim more RAM for the expert cache than the conservative auto-detect |
+| `PIN=stats PIN_GB=g` | pin the hottest experts from a measured usage profile |
+| `DRAFT=n` | MTP draft depth (0 disables speculation) |
+| `GRAMMAR=g.gbnf` | grammar-forced drafts for constrained JSON/NDJSON output ([docs](grammar-draft.md)) |
+| `THINK=1` | enable GLM-5.2's reasoning block |
+| `PILOT=1` | router-lookahead disk prefetch (see below) |
+| `URING=1` | Linux-only batched expert I/O (implies `PIPE=1`) |
+| `PIPE=0` | disable the async expert-load pool (default ON — overlaps `pread` with matmul, −18% disk service) |
+| `DIRECT=1` | O_DIRECT expert reads (measured **+65%** alone on a Strix Halo, [#200](https://github.com/JustVugg/colibri/issues/200)) |
+| `COLI_NUMA=1` | interleave resident weights across NUMA nodes on multi-socket hosts ([#82](https://github.com/JustVugg/colibri/issues/82)) |
+| `CACHE_ROUTE=1` | cache-aware max-rank routing (opt-in, [#199](https://github.com/JustVugg/colibri/issues/199)) |
+| `AUTOPIN=0` | disable the learning cache's auto-pin |
+| `CAP_RAISE=0` | don't auto-grow the expert cache |
+| `KVSAVE=0` | disable KV-cache persistence |
+| `TF=1` | teacher-forcing validation |
+
+## Resource policy
+
+`coli plan` reports the planned hot (VRAM), warm (RAM), and cold backing (disk)
+tiers, the reason for each placement, and the expected bottleneck. The default
+`--policy quality` and `--policy balanced` modes preserve checkpoint quantization
+and router decisions unless `--topk` or `--topp` is passed; those explicit lossy
+overrides print a warning and proceed.
+
+Auto-tier plans size OpenMP from physical cores and bind workers across cores.
+Memory-bound quantized kernels can regress sharply when SMT siblings compete for
+limited memory channels; explicit `OMP_*` settings always take precedence.
+
+```bash
+coli plan --model /models/glm52_i4 --policy quality
+coli run --auto-tier --policy quality "Explain MoE offloading"
+# Explicit research-only router reduction:
+coli run --policy experimental-fast --topk 4 "Benchmark prompt"
+```
+
+Disk is an immutable recovery source, not a normal decode target. If the plan
+leaves cold expert bytes on disk, speed depends on cache hit rate; output quality
+does not.
+
+Cold expert reads can use a deferred pipeline: resident RAM/VRAM experts execute
+while missing experts are loaded in a bounded background I/O pool, then the cold
+results join before the layer completes. The pool engages only under `PIPE=1`;
+`PIPE_WORKERS=n` sets its worker count (default 8). Profiling reports both disk
+service time and the smaller foreground-visible wait time so overlap is explicit.
+
+`--policy balanced` enables lossless live placement (`REPIN=64`). At safe request
+boundaries, a per-layer LFRU score combines decaying session frequency with recent
+access and replaces at most four sufficiently colder pinned experts. `--policy
+quality` leaves live replacement off by default; `REPIN=0` always disables it.
+
+## The learning cache
+
+The engine records which experts your usage actually routes to (`.coli_usage`
+next to the model, updated every turn) and at startup automatically pins the
+hottest ones in spare RAM — colibrì literally gets faster the more you use it.
+`PIN=auto` seeds the pin directly from the live usage history
+([#301](https://github.com/JustVugg/colibri/pull/301)).
+
+**The expert cache auto-sizes to your RAM** (since 2026-07-10): the engine
+*raises* the LRU cap to fill your `--ram` budget instead of only lowering it.
+If you benchmarked colibrì before that date, rerun — your numbers were capped.
+
+**Live tier adaptation** (`--repin N`, opt-in): at safe turn boundaries, a
+decaying session heat map replaces cold pinned experts with hotter streamed
+experts. A 25% hysteresis and a four-swap limit prevent tier thrashing.
+Persistent `.coli_usage` remains the long-term signal and is not decayed.
+
+## Router-lookahead prefetch (`PILOT=1`, experimental)
+
+GLM-5.2's expert routing is measurably predictable *ahead of time* — applying
+layer L+1's router to layer L's post-attention state recalls **71.6%** of the
+true top-8 (vs 41.3% for "same experts as last token"). `PILOT=1` issues
+next-layer expert readahead from a dedicated I/O thread while the current layer
+computes. `PILOT_REAL=1` moves the prefetched loads off the critical path
+(measured +11pp hit rate on a big-cache host), and `PILOT_TWO=1` folds the
+computed shared-expert into the prediction (+3% recall,
+[#200](https://github.com/JustVugg/colibri/issues/200)). On disk-saturated
+hosts hint-only PILOT can be net negative — measure on yours.
+
+## Speculation and reproducibility
+
+Speculative decoding requires that the draft and verify paths compute the same
+function — `SPEC_PIN=1` (default since [#294](https://github.com/JustVugg/colibri/pull/294))
+pins every forward issued while drafts are live to the platform's S=1 kernel
+family. For byte-exact reproducibility across runs: `DRAFT=0`, plus `IDOT=0
+COLI_CUDA=0` if you also want kernel-family/GPU independence. Acceptance
+percentages are not comparable across engine versions under `--topp`
+([#163](https://github.com/JustVugg/colibri/issues/163) has the full story).
+
+## Conversations reopen warm
+
+`coli chat` persists the compressed MLA KV-cache to disk after every turn
+(`.coli_kv`, ~182 KB/token, appended incrementally, crash-safe). Close the chat,
+reopen it tomorrow — the model still remembers the whole conversation and **zero
+re-prefill happens**: validated byte-identical to an uninterrupted session.
+`:reset` clears it, `KVSAVE=0` disables it.

docs/windows.md

@@ -0,0 +1,90 @@
+# Windows 11 (native, no WSL)
+
+colibrì builds and runs natively on Windows 11 x86-64 with MinGW-w64. The port
+adds a `_WIN32` compatibility layer in `c/compat.h` that maps POSIX I/O to the
+Windows API (pread → ReadFile+OVERLAPPED, posix_fadvise no-op, aligned
+allocation, MoveFileEx rename, GlobalMemoryStatusEx RAM detection). All platform
+differences stay in `compat.h`; the engine source is unchanged.
+
+**Toolchain:** GCC via [winlibs](https://winlibs.com/) or MSYS2 MinGW-w64.
+Tested with GCC 16.1.0 (x86_64-ucrt-posix-seh).
+
+```powershell
+# One-time toolchain install (pick one):
+scoop install mingw-winlibs                    # portable, no shell needed
+# or: pacman -S mingw-w64-x86_64-gcc make     # via MSYS2
+
+# Build (from c/ directory):
+make glm.exe            # GLM-5.2 engine (static, no DLL dependencies)
+make olmoe.exe          # OLMoE engine (same shims)
+make iobench.exe        # disk I/O benchmark
+make test-c             # run C tests
+make test-python        # run Python tests (requires python)
+
+# AVX-VNNI: Intel Alder Lake+ (and Meteor Lake+) CPUs have a 128-bit int8
+# dot-product instruction (VPDPBUSD) the engine can use for ~1.3x faster
+# quantized matmul. The x86-64-v3 default (portable AVX2) compiles it out;
+# build for THIS machine to enable it:
+make glm.exe ARCH=native                       # banner prints "idot: avx-vnni"
+
+# Verify (tiny model, 2.4 MB):
+pip install torch transformers safetensors huggingface_hub
+python tools/make_glm_oracle.py                # generate tiny oracle
+SNAP=./glm_tiny TF=1 ./glm.exe 64 16 16        # expect "32/32 positions"
+
+# Run with real model:
+SNAP=D:\glm52_i4 ./glm.exe 64 4 16            # batch inference
+python coli chat --model D:\glm52_i4            # interactive chat
+python coli serve --model D:\glm52_i4            # OpenAI-compatible API
+```
+
+> Windows Store's `python` alias stub is the single most common native-Windows
+> trap: install real Python (python.org or `winget install Python.Python.3.12`)
+> or disable the alias under *Settings → Apps → App execution aliases*.
+
+## Warmup (overnight cache priming)
+
+The engine's expert cache learns from your workload. The included `warmup.ps1`
+script runs `coli run` in a loop with diverse prompts to build the
+`.coli_usage` histogram unattended, so the next real session starts with a
+large, accurate hot-expert pin. Each run saves usage atomically on clean
+completion.
+
+```powershell
+.\warmup.ps1 -Rounds 1 -Ngen 32               # ~60-90 min, durable progress
+```
+
+## NVIDIA GPU (optional, via runtime DLL)
+
+On Windows the engine is built with MinGW gcc but CUDA kernels require MSVC +
+nvcc. The split is clean: build the CUDA backend into a standalone
+`coli_cuda.dll` (nvcc + MSVC), then the host `glm.exe` loads it at runtime via
+`LoadLibrary` (`c/backend_loader.c`). The host never links cudart directly; if
+the DLL is absent the engine falls back to CPU without error.
+
+```powershell
+# Prerequisites: CUDA Toolkit + MSVC Build Tools (cl.exe) + nvcc on PATH.
+# Build the DLL from a shell with the MSVC environment set (vcvars64.bat or
+# "x64 Native Tools Command Prompt for VS"):
+make cuda-dll CUDA_HOME="C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.8" CUDA_ARCH=sm_120
+
+# Build the host with the runtime loader (CUDA_DLL=1 adds -DCOLI_CUDA and
+# links backend_loader.o instead of cudart):
+make glm.exe CUDA_DLL=1 ARCH=native
+
+# Run with the GPU expert tier (8 GB VRAM budget here; scale to your free VRAM):
+$env:COLI_CUDA="1"; $env:COLI_GPU="0"; $env:CUDA_EXPERT_GB="8"
+python coli chat --model D:\glm52_i4 --topp 0.7
+```
+
+The DLL exports the full `extern "C"` surface (including the #111 pipeline ABI);
+`backend_loader.c` resolves symbols via `GetProcAddress` on first use.
+`ColiCudaTensor*` is opaque to the host (stored, never dereferenced), so the
+MSVC-allocated struct is safe across the ABI boundary. `CUDA_ARCH` must match
+your GPU's compute capability (e.g. `sm_120` for Blackwell / RTX 50-series,
+`sm_89` for Ada / RTX 40-series). A one-shot `build_cuda.bat` wrapper is also
+available.
+
+**Measured on a single RTX 5070 Ti + Core Ultra 9 (32 GB RAM):** CPU-only 0.63
+→ CUDA attention+dense 0.72 → **1.07 tok/s** with the GPU-resident pipeline at
+decode ([#273](https://github.com/JustVugg/colibri/issues/273), merged in #274).