diff --git a/README.md b/README.md index 053f9e8..4045e9a 100644 --- a/README.md +++ b/README.md @@ -4,10 +4,10 @@ **Tiny engine, immense model.** Run **GLM-5.2 (744B-parameter MoE)** on a consumer machine with ~25 GB of RAM — in pure C, with zero dependencies, by streaming experts from disk. -Colibrì is a lightweight, quality-preserving MoE runtime that treats VRAM, -RAM, and storage as one managed memory hierarchy. Insufficient fast memory may -reduce speed, but the default policy never silently changes model precision or -router semantics. +Colibrì is a lightweight, quality-preserving MoE runtime that treats VRAM, RAM, +and storage as one managed memory hierarchy. Insufficient fast memory may reduce +speed, but the default policy **never silently changes model precision or router +semantics**. ``` $ ./coli chat @@ -17,14 +17,14 @@ $ ./coli chat ◆ Ciao! 😊 Come posso aiutarti oggi? ``` - ## See it running

colibrì web dashboard — live metrics, hardware panel, expert tiers

-

The web dashboard (./coli web): a 744B model answering at 4+ tok/s end-to-end on 6× RTX 5090 — -with live token metrics, the hardware panel, and the VRAM/RAM/disk expert tiers.

+

The web dashboard (./coli web): a 744B model at 4 tok/s, TTFT 1.6 s, disk 0 — +full expert residency on 6× RTX 5090, with live token metrics, the per-turn time breakdown, +the VRAM/RAM/disk tier bar and the live mini-brain in the corner.

the Brain page — 19,456 experts as a live cortex @@ -33,619 +33,174 @@ with live token metrics, the hardware panel, and the VRAM/RAM/disk expert tiers. brightness is routing heat, and every expert routed in a turn flashes white. Hovering shows the expert's measured topic affinity.

-## Contents - -- [The idea](#the-idea) -- [See it running](#see-it-running) -- [What's implemented](#whats-implemented) -- [Honest numbers](#honest-numbers-wsl2-12-cores-25-gb-ram-nvme-via-vhdx) -- [Download the model](#download-the-model) -- [Web dashboard](#web-dashboard) -- [Got a better machine?](#got-a-better-machine-try-it--heres-what-to-expect) +

+ the Atlas page — the measured expert atlas as a 3-D galaxy +

+

The Atlas page: the measured expert atlas +as a 3-D galaxy — 13,260 characterised experts, 1,041 replicated specialists clustering by topic +(poetry, law, Chinese, SQL…). Position is measured routing affinity, not a learned embedding. Drag to spin.

## The idea -A 744B Mixture-of-Experts model activates only ~40B parameters per token — and only ~11 GB of those change from token to token (the routed experts). So: +A 744B Mixture-of-Experts model activates only ~40B parameters per token — and +only ~11 GB of those change from token to token (the routed experts): -- the **dense part** (attention, shared experts, embeddings — ~17B params) stays **resident in RAM at int4** (~9.9 GB); -- the **19,456 routed experts** (75 MoE layers × 256 experts + the MTP head, ~19 MB each at int4) live **on disk** (~370 GB) and are **streamed on demand**, with a per-layer LRU cache, an optional pinned hot-store, and the OS page cache as a free L2. +

+ only ~5.4% of parameters are active per token +

-The engine is a single C file (`c/glm.c`) plus small headers. No BLAS, no Python at runtime, no GPU required (an opt-in CUDA tier for pinned experts exists — see below). +So the model doesn't need to *fit* in fast memory — it needs to be **placed**: -## What's implemented +- the **dense part** (attention, shared experts, embeddings — ~17B params) stays + **resident in RAM at int4** (~9.9 GB); +- the **19,456 routed experts** (75 MoE layers × 256 + the MTP head, ~19 MB each + at int4) live **on disk** (~370 GB) and are **streamed on demand**, with a + per-layer LRU cache, a learned pinned hot-store, and an optional VRAM tier. -- **Faithful GLM-5.2 (`glm_moe_dsa`) forward** — validated token-exact against a `transformers` oracle (teacher-forcing 32/32, greedy 20/20 on a tiny-random model with the real architecture). -- **MLA attention** (q/kv-LoRA, interleaved partial RoPE) with **compressed KV-cache**: 576 floats/token instead of 32,768 (57× smaller — GLM-5.2 has 64 heads and no GQA). -- **DeepSeek-V3-style sigmoid router** (noaux_tc, routed_scaling_factor), shared expert, first-3-dense layers. -- **Native MTP speculative decoding** — GLM-5.2's own multi-token-prediction head (layer 78) drafts tokens that the main model verifies in one batched forward. **The head must be int8** (the converter does this by default): at int4 draft acceptance collapses to 0–4% and speculation never engages; at int8 it's 39–59% acceptance, **2.2–2.8 tokens/forward** (community-measured, [#8](https://github.com/JustVugg/colibri/issues/8)). Lossless *in exact arithmetic* — but **not byte-identical to non-speculative greedy in practice** ([#100](https://github.com/JustVugg/colibri/issues/100)). This isn't MTP-specific: colibrì's quantized integer kernels are shape-dependent, so any batched (S>1) or GPU forward rounds slightly differently from the single-token path, and int4 GLM-5.2 sits close enough to argmax ties that such a rounding change can flip a token. MTP, the CUDA expert tier, and batched prefill are three different ways to trip the same sensitivity (community-confirmed in #100: swapping only the kernel family forks greedy output on 3/5 prompts, with **zero speculation**). Every emitted token is still the argmax of a *valid* forward — the continuation stays correct — it just isn't the same stream. For byte-exact reproducibility: `DRAFT=0` (no speculation), plus `IDOT=0 COLI_CUDA=0` if you also want kernel-family/GPU independence. Under sampling, rejection sampling keeps the distribution correct. Honest caveat from the same measurement: on a **cold** cache each verified draft routes to extra experts (~660 → ~1100 expert-loads/token), so speculation can be a net *time* loss until the cache/pin warms up. -- **Grammar-forced speculative drafts** (`GRAMMAR=file.gbnf`, [#48](https://github.com/JustVugg/colibri/issues/48)) — on constrained-output workloads (JSON/NDJSON, function calling, structured extraction) the grammar itself is a third draft source: wherever it admits exactly **one** legal byte (braces, quotes, key names, enum bodies), that forced span is tokenized and injected as pre-accepted drafts with ~1.0 acceptance — no draft head, no lookup table, and it engages even with the int4 MTP head from [#8](https://github.com/JustVugg/colibri/issues/8). It never constrains sampling: forced spans are verified in the same batch-union forward as any draft, so a wrong or out-of-sync grammar cannot change the output — worst case is rejected drafts, and an adaptive guard turns the source off below 50% acceptance. Byte-level GBNF subset (literals, char classes, `| ( ) ? * +`, comments); `GRAMMAR_DRAFT=n` caps the forced span per forward (default 24). Composes with `DRAFT`/MTP, which fill the free-text gaps between forced spans. Full reference — mechanism, measured A/Bs, when it pays, prior art: [docs/grammar-draft.md](docs/grammar-draft.md). -- **True sampling** — temperature + nucleus, defaults tuned for int4 reality (0.7 / 0.90; the official 1.0 / 0.95 samples quantization noise from the tail). -- **Integer-dot kernels** (Q8_0-style int8 activations, AVX2 `maddubs`): int8 matmuls 1.4–2.5× faster (119 GFLOP/s measured), int4 1.8× in batch — routing decided per shape by measurement (int4 single-row stays f32: it measured slower). -- **MLA weight absorption** (DeepSeek trick) for decode: no per-token k/v reconstruction — the query absorbs `kv_b`, context is projected after attention. Validated exact: TF 32/32 and generation 20/20 with absorption forced everywhere. -- **Async expert readahead**: while one block of experts is being multiplied, the kernel is already reading the next (`WILLNEED`). -- **Quantization kernels**: int8 / packed int4 / packed int2, per-row scales, AVX2, dequant-on-use. Packing validated bit-identical to the int8 container. -- **DSA sparse attention** — GLM-5.2's lightning indexer, faithful to the reference `glm_moe_dsa` modeling: per-layer top-2048 causal key selection (full/shared indexer layers), auto-detected from the `out-idx-*` weights (`--indexer` converter mode, ~189 MB extracted from the FP8 repo). Validated exact: forcing the selection to keep every key reproduces dense attention token-for-token. `DSA=0` disables, `DSA_TOPK` overrides. -- **KV-cache persistence** — conversations reopen **warm** across engine restarts: serve mode appends the compressed MLA KV to `.coli_kv` after every turn (~182 KB/token, crash-safe) and resumes it at startup with zero re-prefill. Validated byte-identical to an uninterrupted session. `KVSAVE=0` disables. -- **Router-lookahead prefetch** (`PILOT=1`, experimental) — the next layer's routing is 71.6% predictable from the current layer's post-attention state (measured); a dedicated I/O thread prefetches those experts while the current layer computes. -- **Batch-union MoE**: in prefill (and MTP verification), each unique expert of the batch is read once and applied to every position that routes to it. -- **Byte-level BPE tokenizer in C** (GPT-2-style with Unicode-property regex, 320k merges). -- **RAM safety**: the expert cache is auto-sized from `MemAvailable` at startup — an honest peak projection (working set, KV, MTP row, reconstruction buffers) so the kernel OOM-killer never fires. -- **Offline FP8→int4 converter** (`c/tools/convert_fp8_to_int4.py`): downloads one shard at a time (~5 GB), dequants (128×128 block scales), requantizes to the engine's container, deletes the shard — the 756 GB FP8 checkpoint never needs to exist on disk at once. Resumable. +The engine is a single C file (`c/glm.c`) plus small headers. No BLAS, no Python +at runtime, no GPU required. -## Honest numbers (WSL2, 12 cores, 25 GB RAM, NVMe via VHDX) +## How it works -Detailed GPU experiment: [GLM-5.2 on 6x RTX 5090](docs/experiments/glm52-6x5090-2026-07-12.md) — full expert residency across VRAM+RAM reaches 6.84 tok/s single-request decode. +### The per-token path -| metric | value | -|---|---| -| model on disk (int4 container) | ~370 GB | -| resident RAM (dense, int4) | 9.9 GB | -| load time | ~30 s | -| peak RSS during chat | ~20 GB (auto-capped) | -| cold decode cost | ~11 GB disk reads/token (75 layers × 8 experts) | -| disk ceiling (this dev box's drive) | ~1 GB/s → ~0.05–0.1 tok/s cold | -| MTP speculation (int8 head) | 2.2–2.8 tok/forward measured ([#8](https://github.com/JustVugg/colibri/issues/8)) | +

+ route → union → place → overlap → learn +

-This is not fast. It is a 744B frontier-class model **answering correctly on a machine that costs less than one H100 fan**. Warm cache, pinned hot experts and MTP push the useful-response latency down considerably; the physics of the disk does the rest. +Every layer of every token walks the same five steps. The design goal is that +**placement only ever decides speed** — the router's decisions and the weights' +precision are the same whether an expert answered from VRAM or from disk. -### SSD note -Cold starts are heavy on random reads (~11 GB/token), but reads don't meaningfully wear an SSD — colibrì's streaming is read-only. The real concerns under heavy use are (1) **swap traffic** if the system runs out of RAM (writes do wear the drive — keep a sane `--ram` budget; colibrì's auto-budget is designed to stay clear of swap) and (2) **sustained thermals**: hours at full read duty cycle will heat cheaper drives. Monitor drive temperature and health. +### One memory hierarchy instead of one memory requirement -## Download the model +

+ VRAM / RAM / NVMe three-tier expert residency +

-A pre-converted **GLM-5.2 int4** model for colibrì is available on Hugging Face — **use the version with the int8 MTP heads** (matey-0's clone): +The same engine spans the whole range: on a 25 GB laptop everything streams from +disk (slow but correct); on a large host the entire expert set becomes resident +(`CUDA_EXPERT_GB=auto PIN_GB=all`) and disk drops out of the decode path +entirely. Between the tiers sits a **learning cache**: the engine records which +experts *your* workload routes to (`.coli_usage`, updated every turn) and pins +the hottest ones automatically — colibrì literally gets faster the more you use +it. On multi-socket hosts, `COLI_NUMA=1` interleaves the resident weights across +memory controllers ([#82](https://github.com/JustVugg/colibri/issues/82)). + +### Never wait for the disk twice + +Misses are expensive, so the engine spends most of its cleverness avoiding and +overlapping them: each expert's three matrices are stored adjacent and read in +one `pread`; a bounded async I/O pool (`PIPE=1`, default) loads missing experts +while resident ones compute; batched positions read each unique expert once +(**batch-union**); and a router-lookahead thread (`PILOT=1`) prefetches the next +layer's experts — routing is measurably **71.6% predictable one layer ahead**. +On GPUs, the resident pipeline (`COLI_CUDA_PIPE=2`) keeps the residual stream +on-device across layers so the CPU expert loop runs uninterrupted; on Apple +Silicon an experimental [Metal backend](docs/metal.md) does the batched expert +math on the unified-memory GPU. + +### Faithful model, compressed state + +The forward pass is validated **token-exact against a `transformers` oracle** +(teacher-forcing 32/32). MLA attention stores a compressed KV state — 576 +floats/token instead of 32,768 (**57× smaller**) — and persists it across +restarts (`.coli_kv`): conversations reopen warm with zero re-prefill, +byte-identical to an uninterrupted session. DSA sparse attention (GLM-5.2's +lightning indexer) is implemented faithfully and validated by forcing full-key +selection to reproduce dense attention exactly. + +### Speculative decoding, honestly + +GLM-5.2's native MTP head drafts tokens that the main model verifies in one +batched forward — 2.2–2.8 tokens/forward when it pays. Two hard-won rules ship +as defaults: the MTP head must be **int8** (int4 heads collapse to 0–4% +acceptance, [#8](https://github.com/JustVugg/colibri/issues/8)), and draft and +verify must compute **the same function** — `SPEC_PIN=1` pins both to one +kernel family ([#163](https://github.com/JustVugg/colibri/issues/163) is the +full forensic story). Grammar-forced drafts +([`GRAMMAR=file.gbnf`](docs/grammar-draft.md)) add ~free acceptance on +constrained JSON output. Whether speculation is a net win depends on your +cache temperature — measure, and use `DRAFT=0` when it doesn't pay. + +## What it achieves + +

+ measured decode speed by hardware class +

+ +Same engine, same int4 container — the hardware only changes where the experts +live. Highlights from the [full benchmark tables](docs/benchmarks.md): + +- **6× RTX 5090, full residency:** 5.8–6.8 tok/s decode, TTFT ~13 s + ([experiment log](docs/experiments/glm52-6x5090-2026-07-12.md)); +- **128 GB CPU-only desktop:** ~1.8 tok/s warm ([#200](https://github.com/JustVugg/colibri/issues/200)); +- **single RTX 5070 Ti laptop-class box:** 1.07 tok/s via the GPU-resident + pipeline ([#273](https://github.com/JustVugg/colibri/issues/273)); +- **25 GB dev box:** 0.05–0.1 tok/s cold — the proven floor where this project + started, and still the honest baseline. + +Quality is measured, not assumed: the int4 container's quantization cost and the +scale-granularity/rotation ablations live in +[docs/benchmarks.md](docs/benchmarks.md#quality-benchmark) and +[#108](https://github.com/JustVugg/colibri/issues/108)/[#81](https://github.com/JustVugg/colibri/issues/81). + +## Get started + +### 1. Get the model + +A pre-converted **GLM-5.2 int4** container is on Hugging Face — **use the +version with the int8 MTP heads**: **https://huggingface.co/mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp** -> ⚠️ **The MTP head must be int8.** The original mirror ([jlnsrk/GLM-5.2-colibri-int4](https://huggingface.co/jlnsrk/GLM-5.2-colibri-int4)) ships **int4** MTP heads, which give **0% draft acceptance** — speculation silently never engages and you lose the ~2× MTP lever. This is the single most common "why is MTP stuck at 0%?" report ([#8](https://github.com/JustVugg/colibri/issues/8), [#102](https://github.com/JustVugg/colibri/issues/102)). The int8 head gives the measured **39–59% acceptance**. matey-0's clone above is the original int4 model with the three `out-mtp-*` files already swapped to int8 — download that one and you're done. -> -> Check what you have: `ls -l /out-mtp-*` -> · **int8 (correct):** `3527131672 / 5366238584 / 1065950496` -> · **int4 (0% acceptance):** `1765523544 / 2686077736 / 536747200` — if you see these, replace just those three files from the int8 mirror. +> ⚠️ The original mirror ships int4 MTP heads → 0% draft acceptance +> ([#8](https://github.com/JustVugg/colibri/issues/8)). Check yours: +> `ls -l /out-mtp-*` — int8 (correct) is `3527131672 / 5366238584 / 1065950496`. -Download the repository and point `COLI_MODEL` to its directory: +Or convert from the FP8 source yourself — one resumable command that never needs +the full 756 GB on disk at once: ```bash -COLI_MODEL=/path/to/GLM-5.2-colibri-int4-with-int8-mtp ./coli chat +cd c && ./setup.sh # checks gcc/OpenMP, builds, self-tests +./coli convert --model /nvme/glm52_i4 # download+convert shard by shard (python, one-time) ``` -This skips the FP8 → int4 conversion step entirely. Thanks to DatPat for the original mirror and matey-0 for the int8-head clone. - -### Quick start +### 2. Run it ```bash -cd c -./setup.sh # checks gcc/OpenMP, builds, self-tests - -# ONE command does everything model-side: downloads GLM-5.2-FP8 shard by shard -# (never needs the full 756 GB at once), converts to the int4 container, then -# converts the MTP head for speculative decoding. Resumable at any point. -# Conversion (only) needs python with: pip install torch safetensors huggingface_hub numpy -./coli convert --model /nvme/glm52_i4 # ~400 GB free on a real ext4/NVMe path - -# chat — RAM budget, expert cache and MTP are all detected automatically: -COLI_MODEL=/nvme/glm52_i4 ./coli chat +COLI_MODEL=/nvme/glm52_i4 ./coli chat # RAM budget, cache and MTP auto-detected +COLI_MODEL=/nvme/glm52_i4 ./coli plan # inspect the planned VRAM/RAM/disk placement +COLI_MODEL=/nvme/glm52_i4 ./coli doctor # read-only readiness check +./coli web --model /nvme/glm52_i4 # API + web dashboard on one port +./coli serve --model /nvme/glm52_i4 # OpenAI-compatible API only ``` -Inspect the planned storage hierarchy before loading the model: +The engine at runtime is pure C — python is only used by the one-time converter +and the optional API gateway. -```bash -COLI_MODEL=/nvme/glm52_i4 ./coli plan -COLI_MODEL=/nvme/glm52_i4 ./coli plan --gpu 0,1 --ram 128 --vram 48 --json +### 3. Go deeper -# apply the bounded plan to the normal runner -COLI_MODEL=/nvme/glm52_i4 ./coli chat --auto-tier -``` - -`coli plan` reads only safetensors headers and reports the model's exact dense/expert -footprint, runtime RAM reserve, safe expert-cache cap, and bounded VRAM hot tier. Its -versioned JSON output is intended to be shared by the CLI, API server, Web UI, and -desktop shell; it does not allocate model tensors or start inference. -`--auto-tier` applies the same plan to `chat`, `run`, `serve`, and benchmarks. It -sets the RAM budget and context immediately; the VRAM tier is enabled only when -the current `glm` binary is linked with CUDA. Explicit flags and environment -variables keep precedence over automatic values. - -Before loading the model, `coli doctor` performs a read-only readiness check and -explains whether the selected Disk/RAM/VRAM placement is runnable: - -```bash -COLI_MODEL=/nvme/glm52_i4 ./coli doctor -COLI_MODEL=/nvme/glm52_i4 ./coli doctor --gpu 0 --ram 128 --json -``` - -Doctor validates the model directory, config, tokenizer, safetensors headers, -engine executable, available RAM, requested NVIDIA devices, CUDA linkage, and the -same placement budget used by `coli plan`. It never starts `glm`, reads tensor -payloads, imports a model framework, or creates a CUDA context. The versioned JSON -report uses stable check IDs for automation. Warnings keep exit status 0; missing -requirements or an unsafe RAM projection return 1, while invalid CLI values return 2. - -The engine at runtime is pure C — python is only used by the one-time converter. - -### Windows 11 (native, no WSL) - -colibrì builds and runs natively on Windows 11 x86-64 with MinGW-w64. The port adds -a `_WIN32` compatibility layer in `c/compat.h` that maps POSIX I/O to the Windows API -(pread → ReadFile+OVERLAPPED, posix_fadvise no-op, aligned allocation, MoveFileEx rename, -GlobalMemoryStatusEx RAM detection). All platform differences stay in `compat.h`; the -engine source is unchanged. - -**Toolchain:** GCC via [winlibs](https://winlibs.com/) or MSYS2 MinGW-w64. Tested with -GCC 16.1.0 (x86_64-ucrt-posix-seh). - -```powershell -# One-time toolchain install (pick one): -scoop install mingw-winlibs # portable, no shell needed -# or: pacman -S mingw-w64-x86_64-gcc make # via MSYS2 - -# Python (needed by the `coli` CLI and API server). A fresh Windows resolves -# `python` to a Microsoft Store alias stub that opens the Store instead of -# running anything (#198) — installing the real interpreter replaces the stub: -winget install -e --id Python.Python.3.12 - -# Build (from c/ directory): -make glm.exe # GLM-5.2 engine (static, no DLL dependencies) -make olmoe.exe # OLMoE engine (same shims) -make iobench.exe # disk I/O benchmark -make test-c # run C tests -make test-python # run Python tests (requires python) - -# AVX-VNNI: Intel Alder Lake+ (and Meteor Lake+) CPUs have a 128-bit int8 -# dot-product instruction (VPDPBUSD) the engine can use for ~1.3x faster -# quantized matmul. The x86-64-v3 default (portable AVX2) compiles it out; -# build for THIS machine to enable it: -make glm.exe ARCH=native # banner prints "idot: avx-vnni" - -# Verify (tiny model, 2.4 MB): -pip install torch transformers safetensors huggingface_hub -python tools/make_glm_oracle.py # generate tiny oracle -SNAP=./glm_tiny TF=1 ./glm.exe 64 16 16 # expect "32/32 positions" - -# Run with real model: -SNAP=D:\glm52_i4 ./glm.exe 64 4 16 # batch inference -python coli chat --model D:\glm52_i4 # interactive chat -python coli serve --model D:\glm52_i4 # OpenAI-compatible API -``` - -**Warmup (overnight cache priming):** the engine's expert cache learns from -your workload. The included `warmup.ps1` script runs `coli run` in a loop with -diverse prompts to build the `.coli_usage` histogram unattended, so the next -real session starts with a large, accurate hot-expert pin. Each run saves usage -atomically on clean completion. - -```powershell -.\warmup.ps1 -Rounds 1 -Ngen 32 # ~60-90 min, durable progress -``` - -**NVIDIA GPU (optional, via runtime DLL):** on Windows the engine is built with -MinGW gcc but CUDA kernels require MSVC + nvcc. The split is clean: build the -CUDA backend into a standalone `coli_cuda.dll` (nvcc + MSVC), then the host -`glm.exe` loads it at runtime via `LoadLibrary` (`c/backend_loader.c`). The host -never links cudart directly; if the DLL is absent the engine falls back to CPU -without error. - -```powershell -# Prerequisites: CUDA Toolkit + MSVC Build Tools (cl.exe) + nvcc on PATH. -# Build the DLL from a shell with the MSVC environment set (vcvars64.bat or -# "x64 Native Tools Command Prompt for VS"): -make cuda-dll CUDA_HOME="C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.8" CUDA_ARCH=sm_120 - -# Build the host with the runtime loader (CUDA_DLL=1 adds -DCOLI_CUDA and -# links backend_loader.o instead of cudart): -make glm.exe CUDA_DLL=1 ARCH=native - -# Run with the GPU expert tier (8 GB VRAM budget here; scale to your free VRAM): -$env:COLI_CUDA="1"; $env:COLI_GPU="0"; $env:CUDA_EXPERT_GB="8" -python coli chat --model D:\glm52_i4 --topp 0.7 -``` - -The DLL exports 11 `extern "C"` symbols (`coli_cuda_init`, `coli_cuda_matmul`, -etc.); `backend_loader.c` resolves them via `GetProcAddress` on first use. -`ColiCudaTensor*` is opaque to the host (stored, never dereferenced), so the -MSVC-allocated struct is safe across the ABI boundary. `CUDA_ARCH` must match -your GPU's compute capability (e.g. `sm_120` for Blackwell / RTX 50-series, -`sm_89` for Ada / RTX 40-series). - -**Status:** Phase 1 complete (compiles, correct, static-linked). The Windows -GPU tier (runtime `coli_cuda.dll` via `LoadLibrary`) is implemented and -verified on RTX 50-series (sm_120). O_DIRECT (Phase 2) and full-model -validation against the transformers oracle remain separate workstreams. - -### OpenAI-compatible API - -`coli serve` keeps one model process loaded and exposes a text-only OpenAI-compatible -HTTP API. The gateway uses only the Python standard library; inference still runs in -the same dependency-free C engine. - -```bash -cd c -COLI_MODEL=/nvme/glm52_i4 COLI_API_KEY=local-secret ./coli serve \ - --host 127.0.0.1 --port 8000 --model-id glm-5.2-colibri - -curl http://127.0.0.1:8000/v1/chat/completions \ - -H 'Authorization: Bearer local-secret' \ - -H 'Content-Type: application/json' \ - -d '{ - "model": "glm-5.2-colibri", - "messages": [{"role": "user", "content": "Hello"}], - "stream": true - }' -``` - -Implemented endpoints are `GET /v1/models`, `GET /v1/models/{model}`, -`POST /v1/chat/completions`, and legacy `POST /v1/completions`. Chat and -completion requests support JSON responses, SSE streaming, usage counts, -`max_tokens`/`max_completion_tokens`, `temperature`, and `top_p`. The extension -`enable_thinking: true` enables GLM-5.2's reasoning block; the standard -`reasoning_effort` field also enables it unless set to `none`. - -The first version is deliberately text-only and serves one generation at a time: -the 744B model stays in one persistent process, so concurrent HTTP requests queue -instead of loading duplicate model copies. Tools, image/audio input, custom stop -sequences, log probabilities, and token penalties return an explicit error rather -than being silently ignored. The default bind address is localhost; set -`COLI_API_KEY` before exposing the server beyond the machine. - -Browser access from the Vite development server and Tauri local origins is enabled -by default. Repeat `--cors-origin https://your-ui.example` to allow another exact -origin, or use `--cors-origin '*'` only on a trusted local network. - -The engine owns one mutable KV context, so HTTP generation uses a bounded FIFO -admission queue instead of pretending to run unsafe parallel sequences. Configure it -with `--max-queue N` (default 8) and `--queue-timeout SECONDS` (default 300), or the -`COLI_MAX_QUEUE` / `COLI_QUEUE_TIMEOUT` environment variables. Saturated and timed-out -requests receive OpenAI-shaped HTTP 429 errors before streaming headers are sent. -`GET /health` exposes active/queued/completed/rejected counters, and successful -generation responses include `x-colibri-queue-wait-ms`. - -### Isolated KV contexts - -`coli serve --kv-slots N` allocates up to 16 independent sequence contexts. Requests -select one with the optional integer `cache_slot` field; ordinary OpenAI clients omit -it and keep the original slot 0 behavior. - -```json -{ - "model": "glm-5.2-colibri", - "messages": [{"role": "user", "content": "Continue this conversation"}], - "cache_slot": 1 -} -``` - -Each slot owns its token history, compressed MLA/DSA KV memory, MTP window, and -crash-safe persistence file (`.coli_kv`, `.coli_kv.1`, ...). The engine still executes -one sequence at a time; this establishes explicit KV ownership without pretending that -threaded HTTP is continuous batching. RAM admission accounts for every configured slot. -Use `COLI_KV_SLOTS=N` as the environment equivalent. Start with a small value: at the -default 4096-token context, every slot costs hundreds of MB. - -### Experimental Metal backend (Apple Silicon) - -On Apple Silicon the decode profile is matmul-bound, and unified memory removes the -PCIe copy tax that keeps CUDA's streaming experts on the CPU — so colibrì has an -opt-in Metal backend that runs the **routed-expert SwiGLU (batched, zero-copy from -the RAM slabs)**, the **fused decode attention** (full MLA layer in one command -buffer, S≤4), and **prefill's large GEMMs** on the GPU. Token-exact vs the CPU path. - -```bash -cd c -make glm METAL=1 # macOS only; no Xcode needed (shader compiles at runtime) -make metal-test # standalone kernel/attention correctness vs CPU reference -COLI_METAL=1 COLI_MODEL=/path/glm52_i4 ./coli chat --ram 96 -``` - -Measured on an M4 Max (128 GB, warm cache, MTP on): CPU 0.30 → Metal **0.42 tok/s (~1.4×)** -(best config adds `DIRECT=1`; ~3× vs this machine's first cold run). -Key design points: Metal's ~5 ms submit latency makes per-matmul dispatch a loss — -everything is batched into few command buffers per layer, and the resident experts' -GPU work is submitted *before* the missed experts' disk reads so I/O and compute -overlap. `COLI_METAL_GEMM_MIN` tunes the prefill GEMM row threshold (default 16). -Streaming, cache, MTP, DSA and the persistence formats are unchanged; every GPU -path falls back to the CPU per-block on any fault. Numerics are dequant→f32-MAC -(same as the CUDA tier); greedy outputs are byte-identical to the CPU engine. - -### Experimental resident CUDA backend - -colibrì includes an opt-in CUDA backend for model-resident tensors. Streaming -experts deliberately remain on the original CPU path for now: copying an expert -from NVMe to the GPU on every use would only replace the disk bottleneck with a -PCIe bottleneck. Resident quantized tensors are uploaded lazily once and reused. - -```bash -cd c -make cuda-test CUDA=1 # q8/q4/q2/f32 kernel correctness -make CUDA=1 -# optional dense-path experiment (hot experts are configured below) -COLI_CUDA=1 COLI_GPU=0 CUDA_DENSE=1 SNAP=/nvme/glm52_i4 ./glm 64 4 4 -``` - -Requirements: Linux, an NVIDIA driver, and a CUDA Toolkit under -`/usr/local/cuda` (override with `CUDA_HOME=/path/to/cuda`). `CUDA_ARCH=native` -builds for the GPU in the current machine; set an explicit architecture when -cross-compiling. Requesting CUDA with a CPU-only binary, an invalid device, or -an unavailable runtime fails at startup instead of silently falling back. - -The normal `make` build and runtime behavior are unchanged. CUDA defaults to an -expert-only accelerator. `CUDA_DENSE=1` additionally distributes resident -dense/attention projection tensors round-robin across the selected devices; -their projected footprint is reserved before the expert tier is placed. On six -RTX 5090s with a 150 GB expert tier, a warmed two-request/64-token GLM-5.2 run -improved from 1.650 to 2.157 aggregate tok/s (+30.8%) while retaining the full -expert tier. Treat this as an opt-in until the projected dense set and the 2 GB -per-device runtime reserve fit the target GPUs. -A measured `PIN` profile can promote its hottest experts into the persistent -VRAM tier while keeping the rest in RAM: - -```bash -STATS=stats.txt SNAP=/nvme/glm52_i4 ./glm 64 4 4 # collect routing frequencies first -COLI_CUDA=1 COLI_GPU=0 CUDA_EXPERT_GB=16 \ -PIN=stats.txt PIN_GB=160 SNAP=/nvme/glm52_i4 ./glm 64 4 4 -# multi-GPU expert tier, 150 GB total budget across six 32 GB devices -COLI_CUDA=1 COLI_GPUS=0,1,2,3,4,5 CUDA_EXPERT_GB=150 \ -CUDA_DENSE=1 PIN=stats.txt PIN_GB=300 RAM_GB=226 \ -SNAP=/nvme/glm52_i4 ./glm 64 4 4 -# large-RAM host: fill safe VRAM, then keep every remaining expert in RAM -COLI_CUDA=1 COLI_GPUS=0,1,2,3,4,5 CUDA_EXPERT_GB=auto \ -CUDA_DENSE=1 COLI_CUDA_ATTN=1 PIN=stats.txt PIN_GB=all RAM_GB=auto \ -SNAP=/nvme/glm52_i4 ./glm 64 4 4 -``` - -Selected experts are uploaded during startup, so capacity failures occur before -inference and the log reports their exact tensor footprint. The budget is clamped -against free VRAM after reserving the projected dense resident set and 2 GB of -runtime headroom per selected device. With `COLI_GPUS`, `CUDA_EXPERT_GB` is a -total budget across the device set; experts are assigned whole to the -least-loaded device that can hold them. Multi-GPU runs also default to -`PIN_FILL=1`: the measured hot set is placed first, then unused VRAM is filled -with zero-heat experts. `CUDA_RELEASE_HOST=1` (the multi-GPU default) releases -the RAM copy after a successful upload and reloads it from disk only if CUDA -later fails. Set either variable to `0` to restore the conservative behavior. -When host backing is released, placement is disjoint and staged: the hottest -prefix is loaded, uploaded to VRAM, and freed before the next-ranked suffix is -loaded into RAM. `PIN_GB` therefore describes the combined ranked set rather -than duplicate RAM and VRAM copies. On a 256 GB dual-socket host, moving from a -150 GB VRAM + 130 GB RAM placement to 150 GB VRAM + 150 GB RAM raised fixed-token -replay from 1.87 to 2.16 tok/s (+15.7%), reduced expert disk wait from 5.144s to -3.948s, and kept the projected RAM peak below `RAM_GB=226`. The cache cap adjusts -down automatically (54 to 40 in that run) so the larger pinned tier does not exceed -the process budget. Start lower on hosts with less available RAM. - -`CUDA_EXPERT_GB=auto` fills each selected device only up to its measured free -memory minus projected dense tensors and 2 GB of runtime headroom. `PIN_GB=all` -then loads every remaining routed expert into RAM, eliminating decode-time disk -misses when the host budget permits it. The regular `RAM_GB` guard still clamps -the per-layer working cache and rejects unsafe projections; this mode is intended -for dedicated high-memory inference hosts, not desktops running other workloads. -On a dedicated 251 GiB host with six RTX 5090s, this mode selected a 176.7 GB -VRAM expert tier and a 191.3 GB RAM tier (all 19,456 experts resident). The -mode also adapts the VRAM tier every 16 emitted tokens by swapping hot RAM -experts into existing GPU slots. A real 64-token greedy GLM-5.2 generation -measured **6.00 tok/s decode**, up from -2.20 tok/s end-to-end with the earlier 150 GB tier; expert hit rate was 100% -and disk wait was zero. Prompt prefill is reported separately. This is a -host-specific capacity result, not a portable default. - -Text-mode timing reports prefill separately from decode. The decode rate starts -after the prompt KV is built, so it is comparable to `REPLAY` throughput without -hiding time-to-first-token. -MTP speculation defaults off on CUDA because cold draft routes increase expert -traffic; an explicit `DRAFT=n` still overrides the default. - -On six RTX 5090 32 GB cards with GLM-5.2 int4, a 150 GB hot-first tier sustained -0.94 token/s over a 64-token varied prompt (87.8% expert hit rate), and reached -1.64 token/s on a warmed short prompt (99.3% hit rate). The same capacity filled -without routing heat managed only 0.29 token/s, so profile quality matters more -than raw VRAM capacity. These are single-run engineering measurements, not a -portable performance guarantee. - -Current limitations: devices use independent contexts and synchronous -host-staged activation copies—there is no P2P/NCCL dependency yet. Independent -expert groups execute concurrently across devices, but a single expert is not -sharded. The kernels are correctness-first custom kernels rather than -cuBLAS/Tensor Core kernels. - -For a reproducible backend A/B without the full checkpoint, generate the -deterministic 313M-parameter `glm_moe_dsa` fixture and run fixed-token replay: - -```bash -cd c -python tools/make_glm_bench_model.py --output /nvme/colibri-bench-medium --device cuda -python tools/benchmark_cuda_fixture.py --model /nvme/colibri-bench-medium --gpu 0 -``` - -The fixture has random weights and is not a language model. It exists only to -preserve the real MLA/MoE/streaming shapes and compare CPU streaming, dense-only -CUDA, CPU hot-store, and CUDA hot-expert execution with identical replay tokens. - -### Web interface - -`web/` contains a community-contributed browser UI (React + TypeScript, a pure -API client — it never touches the engine directly): - -```bash -cd web -npm ci && npm run dev # then point it at an OpenAI-compatible endpoint -``` - -It speaks the standard OpenAI Chat Completions protocol with SSE streaming, so it -works against the colibrì OpenAI-compatible server (in review, #21) or any other -compatible endpoint. Nothing leaves the endpoint you configure. The terminal -`coli chat` remains the first-class interface. - -Useful knobs (env or flags): `--temp T` token sampling temperature (default 0.7 + nucleus 0.90 — tuned for int4; 0 = greedy), `--topp 0.7` adaptive expert top-p (30–40% less disk), `--ngen N` max tokens per answer (`:more` in chat continues a truncated one), `--repin N` adapt RAM/VRAM hot experts every N emitted tokens, `AUTOPIN=0` disable the learning cache's auto-pin, `THINK=1` enable GLM-5.2's reasoning block, `DRAFT=n` MTP draft depth, `GRAMMAR=g.gbnf` grammar-forced drafts for constrained JSON/NDJSON output (`GRAMMAR_DRAFT=n` caps the forced span), `TF=1` teacher-forcing validation, `PILOT=1` router-lookahead disk prefetch (experimental — see below), `URING=1` Linux-only batched expert I/O (implies `PIPE=1`; also batches `PILOT_REAL`), `PIPE=0` disable the async expert-load pool (**default ON on Windows** — overlaps expert `pread` with the matmul so the CPU isn't idle waiting on the SSD; measured −18% disk service time), `RAM_GB=` claim more RAM for the expert cache than the conservative auto-detect (e.g. `RAM_GB=31` on a 32 GB host raises the cache cap and hit rate measurably), `CAP_RAISE=0` don't auto-grow the expert cache. - -### Resource policy - -`coli plan` reports the planned hot (VRAM), warm (RAM), and cold backing -(disk) tiers, the reason for each placement, and the expected bottleneck. The -default `--policy quality` and `--policy balanced` modes preserve checkpoint -quantization and router decisions unless `--topk` or `--topp` is passed; those -explicit lossy overrides print a warning and proceed. - -Auto-tier plans size OpenMP from physical cores and bind workers across cores. -Memory-bound quantized kernels can regress sharply when SMT siblings compete -for limited memory channels; explicit `OMP_*` settings always take precedence. - -```bash -coli plan --model /models/glm52_i4 --policy quality -coli run --auto-tier --policy quality "Explain MoE offloading" -# Explicit research-only router reduction: -coli run --policy experimental-fast --topk 4 "Benchmark prompt" -``` - -Disk is an immutable recovery source, not a normal decode target. If the plan -leaves cold expert bytes on disk, speed depends on cache hit rate; output -quality does not. - -Cold expert reads can use a deferred pipeline: resident RAM/VRAM experts execute -while missing experts are loaded in a bounded background I/O pool, then the -cold results join before the layer completes. The pool engages only under -`PIPE=1`; `PIPE_WORKERS=n` sets its worker count (default 8). Profiling reports -both disk service time and the smaller foreground-visible wait time so overlap -is explicit rather than credited as unexplained speedup. - -`--policy balanced` enables lossless live placement (`REPIN=64`). At safe -request boundaries, a per-layer LFRU score combines decaying session frequency -with recent access and replaces at most four sufficiently colder pinned -experts. `--policy quality` leaves live replacement off by default; `REPIN=0` -always disables it. Persistent `.coli_usage` history and session-local LFRU -state remain separate. - -For single-token q4 CPU experts, gate and up projections share one OpenMP -dispatch while retaining the same per-row AVX2/NEON arithmetic. This removes -one thread-team launch per RAM expert without activation requantization or a -lower-precision fallback. It is a stepping stone toward a persistent native -CPU expert pool, not a replacement for one. - -**The expert cache auto-sizes to your RAM** (since 2026-07-10): the engine now *raises* the LRU cap to fill your `--ram` budget instead of only lowering it. Before this fix a 128 GB machine ran with the same 8-experts/layer cache as a 16 GB one (issue #12) — **if you benchmarked colibrì before this date, rerun: your numbers were capped.** - -**Router-lookahead prefetch** (`PILOT=1`, experimental): GLM-5.2's expert routing is measurably predictable *ahead of time* — applying layer L+1's router to layer L's post-attention state recalls **71.6%** of the true top-8 (vs 41.3% for "same experts as last token"). `PILOT=1` uses this to issue next-layer expert readahead from a dedicated I/O thread while the current layer computes. On our dev box the disk is already ~80% saturated, so it measures neutral; on machines where compute and disk are balanced (like the Ryzen AI 9 in issue #12: 43% disk / 46% matmul) it should overlap real work — measurements welcome. - -**The learning cache**: the engine records which experts your usage actually routes to (`.coli_usage` next to the model, updated every turn) and at startup automatically pins the hottest ones in spare RAM. colibrì literally gets faster the more you use it. - -**Live tier adaptation** (`--repin N`, opt-in): at safe turn boundaries, a decaying -session heat map replaces cold pinned experts with hotter streamed experts. Replacement -loads the expert from disk into the existing RAM slot; GPU-backed slots immediately -refresh the same VRAM tier budget. A 25% hysteresis and a four-swap limit prevent tier -thrashing. Persistent `.coli_usage` remains the long-term signal and is not decayed. - -**Conversations reopen warm** (`.coli_kv`, since 2026-07-10): `coli chat` persists the compressed MLA KV-cache to disk after every turn (~182 KB/token, appended incrementally, crash-safe). Close the chat, reopen it tomorrow — the model still remembers the whole conversation and **zero re-prefill happens**: validated byte-identical to an uninterrupted session. `:reset` clears it, `KVSAVE=0` disables it. - -## Web dashboard - -One command serves the OpenAI-compatible API **and** the web console on the same port, then opens your browser when the engine is ready: - -```bash -cd web && npm install && npm run build # once -./coli web --model -``` - -What you get: - -- **Chat** with live metrics: a flashing token counter while generating, then tok/s, time-to-first-token, prompt→completion counts and queue wait; -- **Runtime panel**: your hardware (CPU, GPUs + VRAM, RAM, cores), the scheduler, and the live expert-tier bar — how many of the 19,456 experts sit in VRAM / RAM / disk right now; -- **Brain**: the whole model as a 76×256 cortex, one cell per expert. Colour = tier, brightness = routing heat, and the experts routed in each turn flash white and decay — you watch the model think. Hover any cell for its tier, heat and [measured topic affinity](https://github.com/JustVugg/colibri/issues/175) (specialists for code, Chinese, math, law… live in layers 11–22); -- **Profiling**: where each turn's wall time went — I/O wait vs expert matmul vs attention vs LM head — as stacked per-turn bars, plus throughput history, tokens-per-forward batching, and a table of the recent turns. The same phase timers behind the `PROFILE` line, streamed live. - -The dashboard talks to the engine over a few tiny protocol lines (`TIERS`, `EMAP`/`HITS`, `PROF`) and plain JSON endpoints — nothing heavier than the engine itself. - -## Got a better machine? Try it — here's what to expect - -colibrì was built on deliberately humble hardware (12 cores, 25 GB RAM, an older DRAM-less NVMe behind a WSL2 VHDX that measured ~1 GB/s random on *this* drive — note WSL2 VHDX is not inherently slow: a community 5090 box measured 10.5 GB/s O_DIRECT through one, [#101](https://github.com/JustVugg/colibri/issues/101)). **Every one of those constraints is a knob your machine can turn up.** The engine needs: Linux (or WSL2), macOS, or **Windows 11 natively (MinGW-w64)**; gcc with OpenMP, AVX2, ≥16 GB RAM, and the ~370 GB int4 model on a local NVMe (ext4/NTFS — never a network/9p mount). - -**How to test it, in order:** - -```bash -cd c && ./setup.sh # build + architecture self-test (expects 32/32) - -# 1) measure YOUR disk the way the engine uses it (parallel 19 MB random reads): -gcc -O2 -fopenmp iobench.c -o iobench -./iobench /path/to/glm52_i4/out-00069.safetensors 19 64 8 0 # buffered, 8 threads -./iobench /path/to/glm52_i4/out-00069.safetensors 19 64 8 1 # O_DIRECT (bypass cache) -# Caveat (#86): iobench reads a bounded ~1 GB shard, so buffered reads on a big-RAM box -# report the PAGE CACHE, not the disk. Use the O_DIRECT run (arg 1) for a true number, and -# run it on a shard you haven't touched this session (a prior buffered run caches its pages). -# On macOS there is no O_DIRECT — iobench uses F_NOCACHE, which stops *new* caching but can't -# evict pages a prior buffered run already resident-mapped, so a macOS "O_DIRECT" figure right -# after a buffered run still reads cache. Reboot or use a fresh shard for a real cold read. - -# 2) chat; watch the per-turn stats line (tok/s, expert hit-rate, RSS): -COLI_MODEL=/path/to/glm52_i4 ./coli chat - -# 3) record expert usage, then pin the hottest experts in your spare RAM: -STATS=stats.txt ./coli chat -PIN=stats.txt PIN_GB=20 ./coli chat # scale PIN_GB to your free RAM - -# 4) quality benchmarks (MMLU/HellaSwag/ARC): -./coli bench -``` - -**Back-of-envelope predictions** (decode is disk-bound: a cold token costs ~11.4 GB of expert reads; MTP speculation roughly halves the effective cost *once the cache is warm*; RAM turns cold reads into free cache hits): - -| machine | expected | +| topic | doc | |---|---| -| this dev box (WSL2 VHDX, ~1 GB/s, 25 GB RAM) | ~0.05–0.1 tok/s cold — proven baseline | -| native Linux, PCIe4 NVMe (~3–5 GB/s random), 32 GB | ~0.5–1 tok/s | -| PCIe5 NVMe or 2×NVMe RAID0 (~8–12 GB/s), 64 GB (PIN ~40 GB of hot experts) | ~2–4 tok/s | -| 128–256 GB RAM, 12 cores (hot experts cached) | ~2–4 tok/s — matmul-bound: ~80 GFLOP/token vs ~250 GFLOP/s of our AVX2 kernels | -| same RAM + 24–32 cores, or AVX-512/VNNI kernels | ~5–15 tok/s — interactive; kernel work is the multiplier | - -These are estimates, not measurements — if you run colibrì on serious hardware, **please open an issue with your numbers**: real datapoints from better machines are exactly what this project needs next. - -### Community benchmarks (measured) - -Real numbers from real machines, stock build (`setup.sh`, gcc 13), greedy decoding, `--ngen 32`, MTP active: - -| machine | disk (iobench, 19 MB × 64, 8 threads) | config | measured | -|---|---|---|---| -| Intel Core Ultra 7 270K Plus (24 threads) · WSL2 · 24 GB RAM · NVMe VHDX ([#2](https://github.com/JustVugg/colibri/issues/2)) | 1.96 GB/s buffered · 2.74 GB/s O_DIRECT | default | 0.07 tok/s · expert hit 3–4% · RSS 14.1 GB | -| 〃 | 〃 | `--topp 0.7` | **0.11 tok/s** · expert hit 11% · RSS 14.7 GB | -| Apple M5 Max (18 cores) · macOS · 128 GB unified · internal SSD ([#4](https://github.com/JustVugg/colibri/issues/4), [#5](https://github.com/JustVugg/colibri/issues/5)) | ~4 GB/s cold (the 14.2 GB/s reading was cache-influenced — see note) | default, MTP off | **1.06 tok/s** · expert hit 23% · RSS 21.8 GB | -| Apple M5 Max · macOS · 128 GB unified · 2 TB SSD · **Metal backend** ([#72](https://github.com/JustVugg/colibri/pull/72), [#87](https://github.com/JustVugg/colibri/issues/87)) | (macOS O_DIRECT figure unreliable — see note) | Metal on · `--ram 96` · 39.7 GB warm pin · MTP off | **1.83 tok/s** · expert hit 66% · warmed 1.11 → 1.83 over the run | -| 〃 · 46.9 GB pin (2.94M-selection history) · `--ram 110`, 1024-token run ([#103](https://github.com/JustVugg/colibri/issues/103)) | 〃 | Metal on (experts + attention) · MTP off | **2.06 tok/s** · hit 72.5% · coherent output · fastest datapoint yet (still on the pre-rebase Metal branch) | -| Mac Mini M4 Pro · macOS · **48 GB** unified · **Metal backend** ([#107](https://github.com/JustVugg/colibri/issues/107)) | 6.59 GB/s F_NOCACHE (fresh shard) | Metal on · `--ram 38` | **0.30 tok/s** (vs 0.18 CPU-only) — entry Apple Silicon on a third the RAM beats the 32-core 9950X row | -| Epyc 9654 ES · Linux · 4x16GB DDR5-4800-rdimm · Samsung PCIe Gen3 x4 NVME SSD | — | `MTP=1 DIRECT=1` | 0.31 tok/s · expert hit 35% · RSS 21.52 GB | -| Ryzen AI 9 HX 370 (Framework 13) · Arch Linux · 128 GB · WD SN850X, BTRFS zstd ([#12](https://github.com/JustVugg/colibri/issues/12)) | — | int8 MTP head · `--cap 32` · 46.7 GB auto-learned PIN | **0.37 tok/s** · expert hit 66% · MTP acceptance 52% (2.59 tok/fw) · RSS 105 GB | -| Ryzen 9 9950X (32 threads) · Linux · 123 GB · Crucial P3 QLC Gen3 ([#31](https://github.com/JustVugg/colibri/issues/31)) | 1.51 GB/s buffered | default, 2 runs from cold | 0.10 tok/s · hit 53% · profile 66% disk | -| 〃 same machine, model moved to a Samsung 9100 PRO PCIe 5.0 ([#31](https://github.com/JustVugg/colibri/issues/31)) | **8.81 GB/s** O_DIRECT | 〃 (usage history retained) | **0.28 tok/s** · hit 57% · profile flips: 32% disk / **57% matmul** | -| Ryzen AI Max+ 395 (Framework Desktop) · Ubuntu · 128 GB LPDDR5x · Intel Optane 905p PCIe 3.0 ([#39](https://github.com/JustVugg/colibri/issues/39)) | 3.27 GB/s buffered | int8 MTP head · fresh history (pure LRU, auto-raised cap 65) | 0.16 tok/s · hit 57% · profile 49% disk / 47% matmul | -| 〃 five runs later — learned pin 47.6 GB ([#39](https://github.com/JustVugg/colibri/issues/39)) | 〃 | `--temp 0.7 --topp 0.7` | **0.40 tok/s** · hit 71% · fastest non-Apple datapoint | -| Ryzen 7 9800X3D (16T) · WSL2 · 70 GB RAM · Samsung 9100 PRO PCIe 5.0 · RTX 5090 ([#101](https://github.com/JustVugg/colibri/issues/101)) | **10.51 GB/s** O_DIRECT | MTP off · learned pin 24 GB · hit 54% · OMP hot-team on | **0.41 tok/s** · disk-bound (36.5 s disk vs 24.0 s matmul) · **CUDA expert tier ≈ 0%** (AVX-512 CPU matches the 5090) · `--topp 0.7` → **0.52 tok/s** | -| EPYC 7443 (24C/48T, Zen3 AVX2) · Linux · **430 GB RAM** · NVMe RAID-Z1 via TrueNAS VM ([#104](https://github.com/JustVugg/colibri/issues/104)) | ~1 GB/s (VM overhead) | 77.5 GB pin · cap auto-raised to 194/layer · MTP off | **1.00 tok/s** · **hit 98%** · disk eliminated → **RAM-bandwidth + matmul bound** (no AVX-512/VNNI on Zen3) | -| Intel i5-12600K (10C/16T, AVX2) · **native Windows 11, no WSL** · 32 GB · MinGW GCC 16.1 ([#113](https://github.com/JustVugg/colibri/issues/113)) | buffered (no O_DIRECT on MinGW) | int8 MTP head · cold, small-RAM (cap ~2/layer) | **0.08 tok/s** · hit 3.7% · **MTP 57% acceptance** — first native-Windows datapoint, port validated | -| Ryzen 9 9950X3D2 (16C/32T, avx512-vnni) · native Linux · 121 GB · Samsung 9100 PRO **PCIe Gen5** · RTX 5090 (28 GB expert tier, 1475 pinned) ([#120](https://github.com/JustVugg/colibri/issues/120)) | **11.48 GB/s** O_DIRECT | `MTP=0 DIRECT=1 PIPE_WORKERS=16 PREFETCH=1` | **1.23 tok/s** · MTP-off wins disk-bound · fastest x86 datapoint yet | -| Ryzen AI Max+ 395 (Strix Halo, 16C/32T Zen5, avx512-vnni) · Arch Linux · 128 GB unified LPDDR5x · SK hynix P41 PCIe 4.0 ([#124](https://github.com/JustVugg/colibri/issues/124)) | — | `DIRECT=1 PIPE=1 --topp 0.7` · auto-pin | 0.06 cold → **1.10 tok/s** sustained · first Strix Halo / gfx1151 datapoint (unified memory: no discrete VRAM tier) | -| Intel Core Ultra 9 185H (16C/22T, avx-vnni) · **native Windows 11, no WSL** · 32 GB · Crucial P3 QLC NTFS · RTX 5070 Ti (unused) ([#128](https://github.com/JustVugg/colibri/issues/128)) | — | int8 MTP head · **with [#131](https://github.com/JustVugg/colibri/pull/131) (pipe + RAM fixes), warm cache, no GPU** | 0.03 cold → **0.5 tok/s** warm (~7-prompt warmup) · cache-warming on native Windows once the portability blockers are fixed — stock main hung on the `\r\n` READY sentinel before #131 | -| Dell Pro Max GB10 (DGX Spark: Grace 10×X925 + 10×A725, **aarch64 i8mm/sve2**) · Linux · 121 GB unified LPDDR5x · Dell OEM 4 TB NVMe · GB10 sm_121 ([#136](https://github.com/JustVugg/colibri/issues/136)) | **5.58 GB/s** O_DIRECT (NVIDIA-OEM unit in #76 was 10.74 — same platform, different SSD) | int8 MTP head · warm cache | 0.21 cold → **0.50 tok/s** warm · hit 83% · MTP 73% (3.20 tok/fw) · **matmul-bound** (matmul 130 s vs disk 58 s) — unified memory, CUDA placement tier neutral; the lever here is an i8mm compute kernel, not placement | - -Takeaways: with 24 GB of RAM the engine auto-caps the expert cache to 2 slots/layer, so decode stays cold even on a disk 2–2.7× faster than the dev box — **on small-RAM machines the RAM cap, not the disk, is the binding constraint**, exactly as the table above predicts; `--topp 0.7` alone bought a clean 1.6× end-to-end speedup. The M5 Max datapoint lands right on the table's second row: **~1 tok/s of a 744B model on a laptop SSD** — and its 14 GB/s disk shifts the bottleneck back to RAM budget and kernels. The Framework 13 rows are the cache thesis proven end-to-end on one machine: 0.29 → 0.37 tok/s (hit 28% → 66%, speculation finally engaging at 52% acceptance) just by giving the cache its RAM — int8 MTP head + a bigger cap + the learned pin. The cap part is now automatic (cap auto-raise, 2026-07-10). The 9950X pair is the cleanest bottleneck experiment yet — same machine, same history, only the disk swapped: ×5.8 disk bandwidth bought ×2.9 tokens, and the profile **flipped from 66% disk to 57% matmul**. But the crossover depends on the CPU kernel: the 9800X3D row ([#101](https://github.com/JustVugg/colibri/issues/101)) shows that with the OMP hot-team tuning on, the AVX-512 CPU matmul is fast enough that even a **10 GB/s NVMe stays disk-bound** — and there the **CUDA expert tier buys ≈ 0%**, because the CPU already matches the 5090 on expert matmul. The GPU tier earns its VRAM only when the CPU is the weak link, not by default. (Honest correction from #101: an earlier version of that report ran with the OMP tuning off, which manufactured a false matmul-bound crossover and a false +14% for CUDA — neither survived a clean re-run.) - -## Quality benchmark — help wanted - -**First measurement is in** ([#108](https://github.com/JustVugg/colibri/issues/108), thanks dnnspaul): the int4 container scored **62.5% mean acc_norm** on hellaswag/arc/mmlu (0-shot log-likelihood, n=40) — below the 85–95% published for full-precision GLM-5.2, but **the gap is not yet attributable to quantization.** Two confounds sit in the way: (1) 0-shot log-likelihood MC scoring badly underserves a *reasoning* model like GLM-5.2 (it never gets to think), so a large gap is expected even at fp16; (2) n=40 is ±14pp. The **decisive experiment** is the OLMoE fp16-vs-int4 A/B under this same harness (small enough to run both precisions) — that delta *is* the quantization cost with the scoring protocol cancelled out. Until it's run, 62.5% is a datapoint, not a verdict. - -The code is here and ready; one command runs it end to end (it auto-downloads the datasets on first use): - -```bash -cd c -pip install tokenizers datasets # in addition to the convert deps above -./coli bench # hellaswag, arc_challenge, mmlu — 40 questions each -./coli bench hellaswag --limit 200 # one task, more questions -./coli bench mmlu arc_challenge --ram 100 # pick tasks, set a RAM budget -``` - -It prints per-task accuracy (log-likelihood scoring, EleutherAI-harness style). **If you can run the OLMoE fp16-vs-int4 A/B (or a large-n GLM run), please open an issue with the numbers** — it's the measurement that turns 62.5% into either "int4 is fine, scoring artifact" or "quantization is the ceiling, grouped-scale is the priority." +| Benchmarks, community datapoints, quality measurements | [docs/benchmarks.md](docs/benchmarks.md) | +| Tuning knobs, policies, the learning cache, prefetch | [docs/tuning.md](docs/tuning.md) | +| Windows 11 native build (+ CUDA DLL) | [docs/windows.md](docs/windows.md) | +| CUDA backend, VRAM expert tier, full residency | [docs/cuda.md](docs/cuda.md) | +| Apple Silicon Metal backend | [docs/metal.md](docs/metal.md) | +| OpenAI-compatible API, KV slots, web dashboard | [docs/api.md](docs/api.md) | +| Grammar-forced drafts (structured output) | [docs/grammar-draft.md](docs/grammar-draft.md) | +| Environment variable inventory | [docs/ENVIRONMENT.md](docs/ENVIRONMENT.md) | ## Supporting the project -colibrì is a one-person project, written and tested entirely on a 12-core laptop with 25 GB of RAM — the numbers above are the ceiling of what I can measure at home. If this project is useful or interesting to you and you'd like to support its development (better test hardware translates *directly* into a faster engine for everyone: real NVMe scaling data, bigger pinned caches, int2/int3 quality sweeps on real benchmarks), you can: +colibrì started as a one-person project on a 12-core laptop with 25 GB of RAM; +today its numbers come from a community of real machines. If it's useful to you: - ⭐ star the repo and share it; -- 🐛 open issues with benchmark numbers from your hardware; -- 💬 reach out via GitHub issues if you'd like to sponsor development or donate hardware. - -Every contribution, from a datapoint to a disk, moves the ceiling. +- 🐛 open issues with benchmark numbers from your hardware — datapoints move + this project more than anything else; +- 💬 reach out via GitHub issues to sponsor development or donate hardware. ## Repo layout @@ -662,20 +217,20 @@ c/ ├── tools/ offline conversion, fixtures and benchmarks ├── scripts/ long-running conversion helpers └── tests/ dependency-free C and Python tests -web/ browser UI (pure OpenAI-API client, community-maintained) +web/ browser UI (pure OpenAI-API client) desktop/ Tauri v2 desktop shell wrapping the web UI +docs/ reference docs, experiments, media ``` The runtime path intentionally stays flat and readable: `glm.c` plus its small -headers. Auxiliary Python and shell tooling is grouped separately and is never a -runtime dependency of the engine. - -From the repository root, `make`, `make check`, and `make clean` delegate to the -engine Makefile. Existing commands run from `c/` continue to work unchanged. +headers. From the repository root, `make`, `make check`, and `make clean` +delegate to the engine Makefile. ## Why "colibrì" -The hummingbird weighs a few grams, hovers in place, and visits a thousand flowers a day. This engine keeps a 744-billion-parameter giant alive on hummingbird rations: 25 GB of RAM, twelve CPU cores, and a lot of disk patience. +The hummingbird weighs a few grams, hovers in place, and visits a thousand +flowers a day. This engine keeps a 744-billion-parameter giant alive on +hummingbird rations: 25 GB of RAM, twelve CPU cores, and a lot of disk patience. ## License diff --git a/docs/api.md b/docs/api.md new file mode 100644 index 0000000..d3b09f1 --- /dev/null +++ b/docs/api.md @@ -0,0 +1,110 @@ +# OpenAI-compatible API, KV contexts & web UI + +## `coli serve` + +`coli serve` keeps one model process loaded and exposes a text-only +OpenAI-compatible HTTP API. The gateway uses only the Python standard library; +inference still runs in the same dependency-free C engine. + +```bash +cd c +COLI_MODEL=/nvme/glm52_i4 COLI_API_KEY=local-secret ./coli serve \ + --host 127.0.0.1 --port 8000 --model-id glm-5.2-colibri + +curl http://127.0.0.1:8000/v1/chat/completions \ + -H 'Authorization: Bearer local-secret' \ + -H 'Content-Type: application/json' \ + -d '{ + "model": "glm-5.2-colibri", + "messages": [{"role": "user", "content": "Hello"}], + "stream": true + }' +``` + +Implemented endpoints are `GET /v1/models`, `GET /v1/models/{model}`, +`POST /v1/chat/completions`, and legacy `POST /v1/completions`. Chat and +completion requests support JSON responses, SSE streaming, usage counts, +`max_tokens`/`max_completion_tokens`, `temperature`, and `top_p`. The extension +`enable_thinking: true` enables GLM-5.2's reasoning block; the standard +`reasoning_effort` field also enables it unless set to `none`. + +The server is deliberately text-only and serves one generation at a time: the +744B model stays in one persistent process, so concurrent HTTP requests queue +instead of loading duplicate model copies. Tools, image/audio input, custom +stop sequences, log probabilities, and token penalties return an explicit error +rather than being silently ignored. The default bind address is localhost; set +`COLI_API_KEY` before exposing the server beyond the machine. + +Browser access from the Vite development server and Tauri local origins is +enabled by default. Repeat `--cors-origin https://your-ui.example` to allow +another exact origin, or use `--cors-origin '*'` only on a trusted local +network. + +The engine owns its KV contexts, so HTTP generation uses a bounded FIFO +admission queue instead of pretending to run unsafe parallel sequences. +Configure it with `--max-queue N` (default 8) and `--queue-timeout SECONDS` +(default 300), or the `COLI_MAX_QUEUE` / `COLI_QUEUE_TIMEOUT` environment +variables. Saturated and timed-out requests receive OpenAI-shaped HTTP 429 +errors before streaming headers are sent. `GET /health` exposes +active/queued/completed/rejected counters, and successful generation responses +include `x-colibri-queue-wait-ms`. + +## Isolated KV contexts + +`coli serve --kv-slots N` allocates up to 16 independent sequence contexts. +Requests select one with the optional integer `cache_slot` field; ordinary +OpenAI clients omit it and keep the original slot 0 behavior. + +```json +{ + "model": "glm-5.2-colibri", + "messages": [{"role": "user", "content": "Continue this conversation"}], + "cache_slot": 1 +} +``` + +Each slot owns its token history, compressed MLA/DSA KV memory, MTP window, and +crash-safe persistence file (`.coli_kv`, `.coli_kv.1`, ...). The engine matches +each request's tokenized prompt against the slot's history and reuses the common +KV prefix, so stateless HTTP turns keep their cache across requests and even +across engine restarts. Use `COLI_KV_SLOTS=N` as the environment equivalent. +Start small: at the default 4096-token context, every slot costs hundreds of MB. + +## Web dashboard + +One command serves the OpenAI-compatible API **and** the web console on the +same port, then opens your browser when the engine is ready: + +```bash +cd web && npm install && npm run build # once +./coli web --model +``` + +What you get: + +- **Chat** with live metrics: a flashing token counter while generating, then + tok/s, time-to-first-token, prompt→completion counts and queue wait; +- **Runtime panel**: your hardware (CPU, GPUs + VRAM, RAM, cores), the + scheduler, and the live expert-tier bar — how many of the 19,456 experts sit + in VRAM / RAM / disk right now; +- **Brain**: the whole model as a 76×256 cortex, one cell per expert. Colour = + tier, brightness = routing heat, and the experts routed in each turn flash + white and decay — you watch the model think. Hover any cell for its tier, + heat and [measured topic affinity](https://github.com/JustVugg/colibri/issues/175); +- **Atlas**: the measured expert atlas as a 3-D galaxy (publish `experts.json` + from `tools/expert_atlas/analyze.py --web`). + +The dashboard talks to the engine over a small line protocol and plain JSON +endpoints — nothing heavier than the engine itself. `web/` is a pure OpenAI-API +client (React + TypeScript) and also works against any other compatible +endpoint; the terminal `coli chat` remains the first-class interface. + +The layout is responsive down to phone widths, and the sidebar carries the full +telemetry stack — hardware, scheduler, tier bar, per-turn time breakdown, tok/s +trend and per-GPU expert counts: + +

+ the dashboard on a phone-sized viewport +    + the telemetry sidebar +

diff --git a/docs/benchmarks.md b/docs/benchmarks.md new file mode 100644 index 0000000..f9c83f2 --- /dev/null +++ b/docs/benchmarks.md @@ -0,0 +1,140 @@ +# Benchmarks & measured numbers + +Everything on this page is a measurement, not a promise. If you run colibrì on +hardware not listed here, **please open an issue with your numbers** — real +datapoints are what move this project. + +## Reference numbers (the original dev box: WSL2, 12 cores, 25 GB RAM, NVMe via VHDX) + +Detailed GPU experiment: [GLM-5.2 on 6× RTX 5090](experiments/glm52-6x5090-2026-07-12.md) — +full expert residency across VRAM+RAM reaches **6.84 tok/s** single-request decode. + +| metric | value | +|---|---| +| model on disk (int4 container) | ~370 GB | +| resident RAM (dense, int4) | 9.9 GB | +| load time | ~30 s | +| peak RSS during chat | ~20 GB (auto-capped) | +| cold decode cost | ~11 GB disk reads/token (75 layers × 8 experts) | +| disk ceiling (this dev box's drive) | ~1 GB/s → ~0.05–0.1 tok/s cold | +| MTP speculation (int8 head) | 2.2–2.8 tok/forward measured ([#8](https://github.com/JustVugg/colibri/issues/8)) | + +This is not fast. It is a 744B frontier-class model **answering correctly on a +machine that costs less than one H100 fan**. Warm cache, pinned hot experts and +MTP push the useful-response latency down considerably; the physics of the disk +does the rest. + +### SSD note + +Cold starts are heavy on random reads (~11 GB/token), but reads don't +meaningfully wear an SSD — colibrì's streaming is read-only. The real concerns +under heavy use are (1) **swap traffic** if the system runs out of RAM (writes +do wear the drive — keep a sane `--ram` budget; colibrì's auto-budget is designed +to stay clear of swap) and (2) **sustained thermals**: hours at full read duty +cycle will heat cheaper drives. Monitor drive temperature and health. + +## Test your machine, in order + +```bash +cd c && ./setup.sh # build + architecture self-test (expects 32/32) + +# 1) measure YOUR disk the way the engine uses it (parallel 19 MB random reads): +gcc -O2 -fopenmp iobench.c -o iobench +./iobench /path/to/glm52_i4/out-00069.safetensors 19 64 8 0 # buffered, 8 threads +./iobench /path/to/glm52_i4/out-00069.safetensors 19 64 8 1 # O_DIRECT (bypass cache) +# Caveat (#86): iobench reads a bounded ~1 GB shard, so buffered reads on a big-RAM box +# report the PAGE CACHE, not the disk. Use the O_DIRECT run (arg 1) for a true number, and +# run it on a shard you haven't touched this session (a prior buffered run caches its pages). +# On macOS there is no O_DIRECT — iobench uses F_NOCACHE, which stops *new* caching but can't +# evict pages a prior buffered run already resident-mapped, so a macOS "O_DIRECT" figure right +# after a buffered run still reads cache. Reboot or use a fresh shard for a real cold read. + +# 2) chat; watch the per-turn stats line (tok/s, expert hit-rate, RSS): +COLI_MODEL=/path/to/glm52_i4 ./coli chat + +# 3) record expert usage, then pin the hottest experts in your spare RAM: +STATS=stats.txt ./coli chat +PIN=stats.txt PIN_GB=20 ./coli chat # scale PIN_GB to your free RAM + +# 4) quality benchmarks (MMLU/HellaSwag/ARC): +./coli bench +``` + +## Back-of-envelope predictions + +Decode is disk-bound: a cold token costs ~11.4 GB of expert reads; MTP +speculation roughly halves the effective cost *once the cache is warm*; RAM +turns cold reads into free cache hits. + +| machine | expected | +|---|---| +| the dev box (WSL2 VHDX, ~1 GB/s, 25 GB RAM) | ~0.05–0.1 tok/s cold — proven baseline | +| native Linux, PCIe4 NVMe (~3–5 GB/s random), 32 GB | ~0.5–1 tok/s | +| PCIe5 NVMe or 2×NVMe RAID0 (~8–12 GB/s), 64 GB (PIN ~40 GB of hot experts) | ~2–4 tok/s | +| 128–256 GB RAM, 12 cores (hot experts cached) | ~2–4 tok/s — matmul-bound: ~80 GFLOP/token vs ~250 GFLOP/s of our AVX2 kernels | +| same RAM + 24–32 cores, or AVX-512/VNNI kernels | ~5–15 tok/s — interactive; kernel work is the multiplier | + +These are estimates, not measurements. + +## Community benchmarks (measured) + +Real numbers from real machines, stock build (`setup.sh`, gcc 13), greedy decoding, `--ngen 32`, MTP active: + +| machine | disk (iobench, 19 MB × 64, 8 threads) | config | measured | +|---|---|---|---| +| Intel Core Ultra 7 270K Plus (24 threads) · WSL2 · 24 GB RAM · NVMe VHDX ([#2](https://github.com/JustVugg/colibri/issues/2)) | 1.96 GB/s buffered · 2.74 GB/s O_DIRECT | default | 0.07 tok/s · expert hit 3–4% · RSS 14.1 GB | +| 〃 | 〃 | `--topp 0.7` | **0.11 tok/s** · expert hit 11% · RSS 14.7 GB | +| Apple M5 Max (18 cores) · macOS · 128 GB unified · internal SSD ([#4](https://github.com/JustVugg/colibri/issues/4), [#5](https://github.com/JustVugg/colibri/issues/5)) | ~4 GB/s cold (the 14.2 GB/s reading was cache-influenced — see note) | default, MTP off | **1.06 tok/s** · expert hit 23% · RSS 21.8 GB | +| Apple M5 Max · macOS · 128 GB unified · 2 TB SSD · **Metal backend** ([#72](https://github.com/JustVugg/colibri/pull/72), [#87](https://github.com/JustVugg/colibri/issues/87)) | (macOS O_DIRECT figure unreliable — see note) | Metal on · `--ram 96` · 39.7 GB warm pin · MTP off | **1.83 tok/s** · expert hit 66% · warmed 1.11 → 1.83 over the run | +| 〃 · 46.9 GB pin (2.94M-selection history) · `--ram 110`, 1024-token run ([#103](https://github.com/JustVugg/colibri/issues/103)) | 〃 | Metal on (experts + attention) · MTP off | **2.06 tok/s** · hit 72.5% · coherent output | +| Mac Mini M4 Pro · macOS · **48 GB** unified · **Metal backend** ([#107](https://github.com/JustVugg/colibri/issues/107)) | 6.59 GB/s F_NOCACHE (fresh shard) | Metal on · `--ram 38` | **0.30 tok/s** (vs 0.18 CPU-only) | +| Epyc 9654 ES · Linux · 4x16GB DDR5-4800-rdimm · Samsung PCIe Gen3 x4 NVME SSD | — | `MTP=1 DIRECT=1` | 0.31 tok/s · expert hit 35% · RSS 21.52 GB | +| Ryzen AI 9 HX 370 (Framework 13) · Arch Linux · 128 GB · WD SN850X, BTRFS zstd ([#12](https://github.com/JustVugg/colibri/issues/12)) | — | int8 MTP head · `--cap 32` · 46.7 GB auto-learned PIN | **0.37 tok/s** · expert hit 66% · MTP acceptance 52% (2.59 tok/fw) · RSS 105 GB | +| Ryzen 9 9950X (32 threads) · Linux · 123 GB · Crucial P3 QLC Gen3 ([#31](https://github.com/JustVugg/colibri/issues/31)) | 1.51 GB/s buffered | default, 2 runs from cold | 0.10 tok/s · hit 53% · profile 66% disk | +| 〃 same machine, model moved to a Samsung 9100 PRO PCIe 5.0 ([#31](https://github.com/JustVugg/colibri/issues/31)) | **8.81 GB/s** O_DIRECT | 〃 (usage history retained) | **0.28 tok/s** · hit 57% · profile flips: 32% disk / **57% matmul** | +| Ryzen AI Max+ 395 (Framework Desktop) · Ubuntu · 128 GB LPDDR5x · Intel Optane 905p PCIe 3.0 ([#39](https://github.com/JustVugg/colibri/issues/39)) | 3.27 GB/s buffered | int8 MTP head · fresh history (pure LRU, auto-raised cap 65) | 0.16 tok/s · hit 57% · profile 49% disk / 47% matmul | +| 〃 five runs later — learned pin 47.6 GB ([#39](https://github.com/JustVugg/colibri/issues/39)) | 〃 | `--temp 0.7 --topp 0.7` | **0.40 tok/s** · hit 71% | +| Ryzen 7 9800X3D (16T) · WSL2 · 70 GB RAM · Samsung 9100 PRO PCIe 5.0 · RTX 5090 ([#101](https://github.com/JustVugg/colibri/issues/101)) | **10.51 GB/s** O_DIRECT | MTP off · learned pin 24 GB · hit 54% · OMP hot-team on | **0.41 tok/s** · disk-bound (36.5 s disk vs 24.0 s matmul) · **CUDA expert tier ≈ 0%** (AVX-512 CPU matches the 5090) · `--topp 0.7` → **0.52 tok/s** | +| EPYC 7443 (24C/48T, Zen3 AVX2) · Linux · **430 GB RAM** · NVMe RAID-Z1 via TrueNAS VM ([#104](https://github.com/JustVugg/colibri/issues/104)) | ~1 GB/s (VM overhead) | 77.5 GB pin · cap auto-raised to 194/layer · MTP off | **1.00 tok/s** · **hit 98%** · disk eliminated → **RAM-bandwidth + matmul bound** | +| Intel i5-12600K (10C/16T, AVX2) · **native Windows 11, no WSL** · 32 GB · MinGW GCC 16.1 ([#113](https://github.com/JustVugg/colibri/issues/113)) | buffered (no O_DIRECT on MinGW) | int8 MTP head · cold, small-RAM (cap ~2/layer) | **0.08 tok/s** · hit 3.7% · **MTP 57% acceptance** — first native-Windows datapoint | +| Ryzen 9 9950X3D2 (16C/32T, avx512-vnni) · native Linux · 121 GB · Samsung 9100 PRO **PCIe Gen5** · RTX 5090 (28 GB expert tier, 1475 pinned) ([#120](https://github.com/JustVugg/colibri/issues/120)) | **11.48 GB/s** O_DIRECT | `MTP=0 DIRECT=1 PIPE_WORKERS=16 PREFETCH=1` | **1.23 tok/s** | +| Ryzen AI Max+ 395 (Strix Halo, 16C/32T Zen5, avx512-vnni) · Arch Linux · 128 GB unified LPDDR5x · SK hynix P41 PCIe 4.0 ([#124](https://github.com/JustVugg/colibri/issues/124)) | — | `DIRECT=1 PIPE=1 --topp 0.7` · auto-pin | 0.06 cold → **1.10 tok/s** sustained · later **1.83 tok/s** on current dev with `DIRECT=1 PIPE=1 PILOT_REAL=1 PILOT_TWO=1` ([#200](https://github.com/JustVugg/colibri/issues/200)) | +| Intel Core Ultra 9 185H (16C/22T, avx-vnni) · **native Windows 11, no WSL** · 32 GB · Crucial P3 QLC NTFS · RTX 5070 Ti ([#128](https://github.com/JustVugg/colibri/issues/128), [#273](https://github.com/JustVugg/colibri/issues/273)) | — | int8 MTP head · warm cache · GPU-resident pipeline at decode | 0.03 cold → 0.5 warm CPU → **1.07 tok/s** with the pipe2 decode gate (#274) | +| Dell Pro Max GB10 (DGX Spark: Grace, **aarch64 i8mm/sve2**) · Linux · 121 GB unified LPDDR5x · GB10 sm_121 ([#136](https://github.com/JustVugg/colibri/issues/136), [#161](https://github.com/JustVugg/colibri/issues/161)) | **5.58 GB/s** O_DIRECT | int8 MTP head · warm cache | 0.50 tok/s warm · **2.4 tok/s full-k8**, **3.33 tok/s** with `CACHE_ROUTE` (#199) | +| **6 × RTX 5090 · dual Xeon Silver 4510 · 251 GB** (author's rig, [experiment log](experiments/glm52-6x5090-2026-07-12.md)) | NVMe | `CUDA_EXPERT_GB=auto PIN_GB=all` full residency · `COLI_CUDA_PIPE=2 TC_W4A16` · DRAFT=0 | **5.8–6.8 tok/s** decode · TTFT ~13 s · hit 89–100% | + +### Takeaways + +With 24 GB of RAM the engine auto-caps the expert cache to 2 slots/layer, so +decode stays cold even on a fast disk — **on small-RAM machines the RAM cap, not +the disk, is the binding constraint**; `--topp 0.7` alone bought a clean 1.6× +end-to-end speedup. The 9950X pair is the cleanest bottleneck experiment: same +machine, same history, only the disk swapped — ×5.8 disk bandwidth bought ×2.9 +tokens, and the profile **flipped from 66% disk to 57% matmul**. But the +crossover depends on the CPU kernel: with OMP hot-team tuning on, an AVX-512 CPU +can match an RTX 5090 on expert matmul ([#101](https://github.com/JustVugg/colibri/issues/101)), +so **the GPU tier earns its VRAM only when the CPU is the weak link**. On +multi-socket hosts, NUMA placement is a further lever: interleaving the resident +weights across nodes measured **+13% (2-socket) and +40% (4-socket CPU-only)** +([#82](https://github.com/JustVugg/colibri/issues/82)) — but never blanket-interleave +a GPU host (measured 10× regression via the DMA staging pages). + +## Quality benchmark + +**Measured** ([#108](https://github.com/JustVugg/colibri/issues/108)): the int4 +container scored **62.5% mean acc_norm** on hellaswag/arc/mmlu (0-shot +log-likelihood, n=40) — but 0-shot MC scoring underserves a reasoning model, and +the OLMoE fp16-vs-int4 A/B under the same harness measured the pure quantization +cost at **-8.2pp**, concentrated on the hardest task (per-row int4 scales erode +the small logit margins hard questions depend on — grouped scales recover ~63% +of that loss, see [#225](https://github.com/JustVugg/colibri/issues/225)). The +scale-granularity/rotation/lattice ablation lives in +`tools/quant_ablation.py` ([#81](https://github.com/JustVugg/colibri/issues/81)). + +```bash +cd c +pip install tokenizers datasets +./coli bench # hellaswag, arc_challenge, mmlu — 40 questions each +./coli bench hellaswag --limit 200 # one task, more questions +./coli bench mmlu arc_challenge --ram 100 # pick tasks, set a RAM budget +``` diff --git a/docs/cuda.md b/docs/cuda.md new file mode 100644 index 0000000..fab477d --- /dev/null +++ b/docs/cuda.md @@ -0,0 +1,104 @@ +# CUDA backend (Linux) + +colibrì includes an opt-in CUDA backend for model-resident tensors. Streaming +experts deliberately remain on the original CPU path: copying an expert from +NVMe to the GPU on every use would only replace the disk bottleneck with a PCIe +bottleneck. Resident quantized tensors are uploaded lazily once and reused. + +```bash +cd c +make cuda-test CUDA=1 # q8/q4/q2/f32 kernel correctness +make CUDA=1 +# optional dense-path experiment (hot experts are configured below) +COLI_CUDA=1 COLI_GPU=0 CUDA_DENSE=1 SNAP=/nvme/glm52_i4 ./glm 64 4 4 +``` + +Requirements: Linux, an NVIDIA driver, and a CUDA Toolkit under +`/usr/local/cuda` (override with `CUDA_HOME=/path/to/cuda`). +`CUDA_ARCH=native` builds for the GPU in the current machine. Requesting CUDA +with a CPU-only binary, an invalid device, or an unavailable runtime fails at +startup instead of silently falling back. For Windows, see +[windows.md](windows.md) (runtime DLL path). + +## The VRAM expert tier + +A measured `PIN` profile promotes its hottest experts into a persistent VRAM +tier while keeping the rest in RAM: + +```bash +STATS=stats.txt SNAP=/nvme/glm52_i4 ./glm 64 4 4 # collect routing frequencies first +COLI_CUDA=1 COLI_GPU=0 CUDA_EXPERT_GB=16 \ +PIN=stats.txt PIN_GB=160 SNAP=/nvme/glm52_i4 ./glm 64 4 4 + +# multi-GPU expert tier, 150 GB total budget across six 32 GB devices +COLI_CUDA=1 COLI_GPUS=0,1,2,3,4,5 CUDA_EXPERT_GB=150 \ +CUDA_DENSE=1 PIN=stats.txt PIN_GB=300 RAM_GB=226 \ +SNAP=/nvme/glm52_i4 ./glm 64 4 4 + +# large-RAM host: fill safe VRAM, then keep every remaining expert in RAM +COLI_CUDA=1 COLI_GPUS=0,1,2,3,4,5 CUDA_EXPERT_GB=auto \ +CUDA_DENSE=1 COLI_CUDA_ATTN=1 PIN=stats.txt PIN_GB=all RAM_GB=auto \ +SNAP=/nvme/glm52_i4 ./glm 64 4 4 +``` + +Selected experts are uploaded during startup, so capacity failures occur before +inference. The budget is clamped against free VRAM after reserving the projected +dense resident set and 2 GB of runtime headroom per device. With `COLI_GPUS`, +`CUDA_EXPERT_GB` is a total budget across the device set; experts are assigned +whole to the least-loaded device that can hold them. Multi-GPU runs default to +`PIN_FILL=1` (measured hot set first, then unused VRAM filled with zero-heat +experts) and `CUDA_RELEASE_HOST=1` (RAM copy released after upload, reloaded +from disk only if CUDA later fails). + +`CUDA_EXPERT_GB=auto` fills each device up to measured free memory minus +projected dense tensors and headroom. `PIN_GB=all` then loads the remaining +routed experts into RAM **up to the `--ram` budget** (it clamps — [#229](https://github.com/JustVugg/colibri/issues/229)), +eliminating decode-time disk misses when capacity permits. This mode is intended +for dedicated high-memory inference hosts. + +### Full-residency reference result (6× RTX 5090, 251 GiB host) + +`CUDA_EXPERT_GB=auto PIN_GB=all` selected a 176.7 GB VRAM tier + 191.3 GB RAM +tier (all 19,456 experts resident), adapting the VRAM tier every 16 tokens. +With the GPU-resident pipeline (`COLI_CUDA_PIPE=2`) and Tensor-Core W4A16 +dispatch (`COLI_CUDA_TC_W4A16=1`), 96-token greedy decode measured +**5.8–6.8 tok/s** (TTFT ~13 s; 1571-token prefill ~122 s then 4.2 tok/s). +Full experiment log: [experiments/glm52-6x5090-2026-07-12.md](experiments/glm52-6x5090-2026-07-12.md). +These are host-specific capacity results, not portable defaults. + +## The GPU-resident pipeline (`COLI_CUDA_PIPE`) + +`COLI_CUDA_PIPE=2` keeps the residual stream on-device across layers: rmsnorms, +residual adds, router GEMMs and the shared expert run on the GPU while the CPU +expert loop runs uninterrupted, with batched attention and grouped expert +uploads at prefill. On a single-GPU host this also pays at decode (S=1): +**+49%** measured on a 5070 Ti ([#273](https://github.com/JustVugg/colibri/issues/273)/#274); +on multi-GPU hosts the per-layer P2P hops cancel the gain, so the decode gate is +device-count aware. `COLI_CUDA_TC_W4A16=1` enables Tensor-Core int4×fp16 mixed +dispatch for batched rows (pays at ≥16 rows). + +## Notes and limitations + +- Text-mode timing reports prefill separately from decode. +- MTP speculation defaults off on CUDA (cold draft routes increase expert + traffic); explicit `DRAFT=n` overrides. Since #294, `SPEC_PIN=1` keeps + draft/verify kernels consistent when speculation is on. +- Devices use independent contexts; a single expert is not sharded. Kernels are + correctness-first custom kernels. +- Profile quality matters more than raw VRAM capacity: the same 150 GB tier + measured 0.94–1.64 tok/s hot-first vs 0.29 tok/s filled without routing heat. +- The GPU tier earns its VRAM only when the CPU is the weak link — a tuned + AVX-512 CPU can match a 5090 on expert matmul + ([#101](https://github.com/JustVugg/colibri/issues/101)). + +## Reproducible backend A/B without the full checkpoint + +```bash +cd c +python tools/make_glm_bench_model.py --output /nvme/colibri-bench-medium --device cuda +python tools/benchmark_cuda_fixture.py --model /nvme/colibri-bench-medium --gpu 0 +``` + +The 313M-parameter fixture has random weights and is not a language model. It +preserves the real MLA/MoE/streaming shapes to compare CPU streaming, dense-only +CUDA, CPU hot-store, and CUDA hot-expert execution with identical replay tokens. diff --git a/docs/media/colibri-atlas.png b/docs/media/colibri-atlas.png new file mode 100644 index 0000000..5e5e76e Binary files /dev/null and b/docs/media/colibri-atlas.png differ diff --git a/docs/media/colibri-brain.png b/docs/media/colibri-brain.png index 0cd6c90..9ac5676 100644 Binary files a/docs/media/colibri-brain.png and b/docs/media/colibri-brain.png differ diff --git a/docs/media/colibri-dashboard.png b/docs/media/colibri-dashboard.png index 123fd3e..d51aa08 100644 Binary files a/docs/media/colibri-dashboard.png and b/docs/media/colibri-dashboard.png differ diff --git a/docs/media/colibri-metrics.png b/docs/media/colibri-metrics.png new file mode 100644 index 0000000..853d686 Binary files /dev/null and b/docs/media/colibri-metrics.png differ diff --git a/docs/media/colibri-mobile.png b/docs/media/colibri-mobile.png new file mode 100644 index 0000000..5c1c63c Binary files /dev/null and b/docs/media/colibri-mobile.png differ diff --git a/docs/media/ladder.png b/docs/media/ladder.png new file mode 100644 index 0000000..cd272fc Binary files /dev/null and b/docs/media/ladder.png differ diff --git a/docs/media/sparse.png b/docs/media/sparse.png new file mode 100644 index 0000000..0d7a4a1 Binary files /dev/null and b/docs/media/sparse.png differ diff --git a/docs/media/tiers.png b/docs/media/tiers.png new file mode 100644 index 0000000..9452587 Binary files /dev/null and b/docs/media/tiers.png differ diff --git a/docs/media/token-path.png b/docs/media/token-path.png new file mode 100644 index 0000000..4aebd39 Binary files /dev/null and b/docs/media/token-path.png differ diff --git a/docs/metal.md b/docs/metal.md new file mode 100644 index 0000000..3c94e5c --- /dev/null +++ b/docs/metal.md @@ -0,0 +1,30 @@ +# Metal backend (Apple Silicon, experimental) + +On Apple Silicon the decode profile is matmul-bound, and unified memory removes +the PCIe copy tax that keeps CUDA's streaming experts on the CPU — so colibrì +has an opt-in Metal backend that runs the **routed-expert SwiGLU (batched, +zero-copy from the RAM slabs)**, the **fused decode attention** (full MLA layer +in one command buffer, S≤4), and **prefill's large GEMMs** on the GPU. +Token-exact vs the CPU path. + +```bash +cd c +make glm METAL=1 # macOS only; no Xcode needed (shader compiles at runtime) +make metal-test # standalone kernel/attention correctness vs CPU reference +COLI_METAL=1 COLI_MODEL=/path/glm52_i4 ./coli chat --ram 96 +``` + +Measured on an M4 Max (128 GB, warm cache, MTP on): CPU 0.30 → Metal +**0.42 tok/s (~1.4×)** (best config adds `DIRECT=1`; ~3× vs this machine's +first cold run). An M5 Max with a 46.9 GB learned pin reached **2.06 tok/s** +([#103](https://github.com/JustVugg/colibri/issues/103); see also the +[M5 Max performance report](METAL-M5MAX-PERF-REPORT.md)). + +Key design points: Metal's ~5 ms submit latency makes per-matmul dispatch a +loss — everything is batched into few command buffers per layer, and the +resident experts' GPU work is submitted *before* the missed experts' disk reads +so I/O and compute overlap. `COLI_METAL_GEMM_MIN` tunes the prefill GEMM row +threshold (default 16). Streaming, cache, MTP, DSA and the persistence formats +are unchanged; every GPU path falls back to the CPU per-block on any fault. +Numerics are dequant→f32-MAC (same as the CUDA tier); greedy outputs are +byte-identical to the CPU engine. diff --git a/docs/tuning.md b/docs/tuning.md new file mode 100644 index 0000000..c95eca8 --- /dev/null +++ b/docs/tuning.md @@ -0,0 +1,110 @@ +# Tuning & runtime knobs + +Everything here is opt-in; the defaults are chosen so a plain `./coli chat` +is safe on any machine. See also [SETTINGS.md](SETTINGS.md) and +[ENVIRONMENT.md](ENVIRONMENT.md) for the full variable inventory. + +## The knobs that matter most + +| knob | what it does | +|---|---| +| `--temp T` | token sampling temperature (default 0.7 + nucleus 0.90 — tuned for int4; 0 = greedy) | +| `--topp 0.7` | adaptive expert top-p (30–40% less disk; lossy — prints a warning) | +| `--ngen N` | max tokens per answer (`:more` in chat continues a truncated one) | +| `--repin N` | adapt RAM/VRAM hot experts every N emitted tokens | +| `RAM_GB=` | claim more RAM for the expert cache than the conservative auto-detect | +| `PIN=stats PIN_GB=g` | pin the hottest experts from a measured usage profile | +| `DRAFT=n` | MTP draft depth (0 disables speculation) | +| `GRAMMAR=g.gbnf` | grammar-forced drafts for constrained JSON/NDJSON output ([docs](grammar-draft.md)) | +| `THINK=1` | enable GLM-5.2's reasoning block | +| `PILOT=1` | router-lookahead disk prefetch (see below) | +| `URING=1` | Linux-only batched expert I/O (implies `PIPE=1`) | +| `PIPE=0` | disable the async expert-load pool (default ON — overlaps `pread` with matmul, −18% disk service) | +| `DIRECT=1` | O_DIRECT expert reads (measured **+65%** alone on a Strix Halo, [#200](https://github.com/JustVugg/colibri/issues/200)) | +| `COLI_NUMA=1` | interleave resident weights across NUMA nodes on multi-socket hosts ([#82](https://github.com/JustVugg/colibri/issues/82)) | +| `CACHE_ROUTE=1` | cache-aware max-rank routing (opt-in, [#199](https://github.com/JustVugg/colibri/issues/199)) | +| `AUTOPIN=0` | disable the learning cache's auto-pin | +| `CAP_RAISE=0` | don't auto-grow the expert cache | +| `KVSAVE=0` | disable KV-cache persistence | +| `TF=1` | teacher-forcing validation | + +## Resource policy + +`coli plan` reports the planned hot (VRAM), warm (RAM), and cold backing (disk) +tiers, the reason for each placement, and the expected bottleneck. The default +`--policy quality` and `--policy balanced` modes preserve checkpoint quantization +and router decisions unless `--topk` or `--topp` is passed; those explicit lossy +overrides print a warning and proceed. + +Auto-tier plans size OpenMP from physical cores and bind workers across cores. +Memory-bound quantized kernels can regress sharply when SMT siblings compete for +limited memory channels; explicit `OMP_*` settings always take precedence. + +```bash +coli plan --model /models/glm52_i4 --policy quality +coli run --auto-tier --policy quality "Explain MoE offloading" +# Explicit research-only router reduction: +coli run --policy experimental-fast --topk 4 "Benchmark prompt" +``` + +Disk is an immutable recovery source, not a normal decode target. If the plan +leaves cold expert bytes on disk, speed depends on cache hit rate; output quality +does not. + +Cold expert reads can use a deferred pipeline: resident RAM/VRAM experts execute +while missing experts are loaded in a bounded background I/O pool, then the cold +results join before the layer completes. The pool engages only under `PIPE=1`; +`PIPE_WORKERS=n` sets its worker count (default 8). Profiling reports both disk +service time and the smaller foreground-visible wait time so overlap is explicit. + +`--policy balanced` enables lossless live placement (`REPIN=64`). At safe request +boundaries, a per-layer LFRU score combines decaying session frequency with recent +access and replaces at most four sufficiently colder pinned experts. `--policy +quality` leaves live replacement off by default; `REPIN=0` always disables it. + +## The learning cache + +The engine records which experts your usage actually routes to (`.coli_usage` +next to the model, updated every turn) and at startup automatically pins the +hottest ones in spare RAM — colibrì literally gets faster the more you use it. +`PIN=auto` seeds the pin directly from the live usage history +([#301](https://github.com/JustVugg/colibri/pull/301)). + +**The expert cache auto-sizes to your RAM** (since 2026-07-10): the engine +*raises* the LRU cap to fill your `--ram` budget instead of only lowering it. +If you benchmarked colibrì before that date, rerun — your numbers were capped. + +**Live tier adaptation** (`--repin N`, opt-in): at safe turn boundaries, a +decaying session heat map replaces cold pinned experts with hotter streamed +experts. A 25% hysteresis and a four-swap limit prevent tier thrashing. +Persistent `.coli_usage` remains the long-term signal and is not decayed. + +## Router-lookahead prefetch (`PILOT=1`, experimental) + +GLM-5.2's expert routing is measurably predictable *ahead of time* — applying +layer L+1's router to layer L's post-attention state recalls **71.6%** of the +true top-8 (vs 41.3% for "same experts as last token"). `PILOT=1` issues +next-layer expert readahead from a dedicated I/O thread while the current layer +computes. `PILOT_REAL=1` moves the prefetched loads off the critical path +(measured +11pp hit rate on a big-cache host), and `PILOT_TWO=1` folds the +computed shared-expert into the prediction (+3% recall, +[#200](https://github.com/JustVugg/colibri/issues/200)). On disk-saturated +hosts hint-only PILOT can be net negative — measure on yours. + +## Speculation and reproducibility + +Speculative decoding requires that the draft and verify paths compute the same +function — `SPEC_PIN=1` (default since [#294](https://github.com/JustVugg/colibri/pull/294)) +pins every forward issued while drafts are live to the platform's S=1 kernel +family. For byte-exact reproducibility across runs: `DRAFT=0`, plus `IDOT=0 +COLI_CUDA=0` if you also want kernel-family/GPU independence. Acceptance +percentages are not comparable across engine versions under `--topp` +([#163](https://github.com/JustVugg/colibri/issues/163) has the full story). + +## Conversations reopen warm + +`coli chat` persists the compressed MLA KV-cache to disk after every turn +(`.coli_kv`, ~182 KB/token, appended incrementally, crash-safe). Close the chat, +reopen it tomorrow — the model still remembers the whole conversation and **zero +re-prefill happens**: validated byte-identical to an uninterrupted session. +`:reset` clears it, `KVSAVE=0` disables it. diff --git a/docs/windows.md b/docs/windows.md new file mode 100644 index 0000000..94a4f69 --- /dev/null +++ b/docs/windows.md @@ -0,0 +1,90 @@ +# Windows 11 (native, no WSL) + +colibrì builds and runs natively on Windows 11 x86-64 with MinGW-w64. The port +adds a `_WIN32` compatibility layer in `c/compat.h` that maps POSIX I/O to the +Windows API (pread → ReadFile+OVERLAPPED, posix_fadvise no-op, aligned +allocation, MoveFileEx rename, GlobalMemoryStatusEx RAM detection). All platform +differences stay in `compat.h`; the engine source is unchanged. + +**Toolchain:** GCC via [winlibs](https://winlibs.com/) or MSYS2 MinGW-w64. +Tested with GCC 16.1.0 (x86_64-ucrt-posix-seh). + +```powershell +# One-time toolchain install (pick one): +scoop install mingw-winlibs # portable, no shell needed +# or: pacman -S mingw-w64-x86_64-gcc make # via MSYS2 + +# Build (from c/ directory): +make glm.exe # GLM-5.2 engine (static, no DLL dependencies) +make olmoe.exe # OLMoE engine (same shims) +make iobench.exe # disk I/O benchmark +make test-c # run C tests +make test-python # run Python tests (requires python) + +# AVX-VNNI: Intel Alder Lake+ (and Meteor Lake+) CPUs have a 128-bit int8 +# dot-product instruction (VPDPBUSD) the engine can use for ~1.3x faster +# quantized matmul. The x86-64-v3 default (portable AVX2) compiles it out; +# build for THIS machine to enable it: +make glm.exe ARCH=native # banner prints "idot: avx-vnni" + +# Verify (tiny model, 2.4 MB): +pip install torch transformers safetensors huggingface_hub +python tools/make_glm_oracle.py # generate tiny oracle +SNAP=./glm_tiny TF=1 ./glm.exe 64 16 16 # expect "32/32 positions" + +# Run with real model: +SNAP=D:\glm52_i4 ./glm.exe 64 4 16 # batch inference +python coli chat --model D:\glm52_i4 # interactive chat +python coli serve --model D:\glm52_i4 # OpenAI-compatible API +``` + +> Windows Store's `python` alias stub is the single most common native-Windows +> trap: install real Python (python.org or `winget install Python.Python.3.12`) +> or disable the alias under *Settings → Apps → App execution aliases*. + +## Warmup (overnight cache priming) + +The engine's expert cache learns from your workload. The included `warmup.ps1` +script runs `coli run` in a loop with diverse prompts to build the +`.coli_usage` histogram unattended, so the next real session starts with a +large, accurate hot-expert pin. Each run saves usage atomically on clean +completion. + +```powershell +.\warmup.ps1 -Rounds 1 -Ngen 32 # ~60-90 min, durable progress +``` + +## NVIDIA GPU (optional, via runtime DLL) + +On Windows the engine is built with MinGW gcc but CUDA kernels require MSVC + +nvcc. The split is clean: build the CUDA backend into a standalone +`coli_cuda.dll` (nvcc + MSVC), then the host `glm.exe` loads it at runtime via +`LoadLibrary` (`c/backend_loader.c`). The host never links cudart directly; if +the DLL is absent the engine falls back to CPU without error. + +```powershell +# Prerequisites: CUDA Toolkit + MSVC Build Tools (cl.exe) + nvcc on PATH. +# Build the DLL from a shell with the MSVC environment set (vcvars64.bat or +# "x64 Native Tools Command Prompt for VS"): +make cuda-dll CUDA_HOME="C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.8" CUDA_ARCH=sm_120 + +# Build the host with the runtime loader (CUDA_DLL=1 adds -DCOLI_CUDA and +# links backend_loader.o instead of cudart): +make glm.exe CUDA_DLL=1 ARCH=native + +# Run with the GPU expert tier (8 GB VRAM budget here; scale to your free VRAM): +$env:COLI_CUDA="1"; $env:COLI_GPU="0"; $env:CUDA_EXPERT_GB="8" +python coli chat --model D:\glm52_i4 --topp 0.7 +``` + +The DLL exports the full `extern "C"` surface (including the #111 pipeline ABI); +`backend_loader.c` resolves symbols via `GetProcAddress` on first use. +`ColiCudaTensor*` is opaque to the host (stored, never dereferenced), so the +MSVC-allocated struct is safe across the ABI boundary. `CUDA_ARCH` must match +your GPU's compute capability (e.g. `sm_120` for Blackwell / RTX 50-series, +`sm_89` for Ada / RTX 40-series). A one-shot `build_cuda.bat` wrapper is also +available. + +**Measured on a single RTX 5070 Ti + Core Ultra 9 (32 GB RAM):** CPU-only 0.63 +→ CUDA attention+dense 0.72 → **1.07 tok/s** with the GPU-resident pipeline at +decode ([#273](https://github.com/JustVugg/colibri/issues/273), merged in #274).