4 Commits

Author SHA1 Message Date
JustVugg cfcc742591 metal: advance the LFRU recency clock on the GPU-prerouted decode path (#417)
On Metal, when routing is precomputed on the GPU (g_pre_idx), the moe fast
path bumps eusage/ehit/eheat for the selected experts but skips the one thing
the full CPU router does at the equivalent site: elast[layer][e] =
++eaccess_clock. So the session-local recency clock advances during prefill
(full router) but freezes the moment GPU-prerouted decode starts, and REPIN's
tier_pick_lfru() tie-breaker then runs on stale recency for the rest of the
run. Mirror the exact update the non-Metal path already does. Inside
#ifdef COLI_METAL, so CPU/CUDA are untouched; elast only feeds the LFRU
eviction heuristic, so this cannot affect output, only which experts REPIN
keeps warm.

Found and reported by @monotophic with a source-level trace repro
(ELAST_TRACE). Fix is inspection-verified against line ~3055; needs a
Metal build to exercise end-to-end.

Closes #417

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 14:58:32 +02:00
Vincenzo Fornaro ebc851edb3 Merge pull request #415 from JustVugg/docs-quickstart
docs: beginner-friendly Quick Start guide (#414)
2026-07-19 14:25:24 +02:00
JustVugg 845af6378d docs: beginner-friendly Quick Start guide for Linux/Windows/macOS (#414)
Adds docs/quickstart.md — a step-by-step, no-experience-assumed walkthrough
from installing the build tools to the first coli chat, with per-OS
copy-paste commands (Ubuntu apt, Windows MSYS2 or prebuilt binary, macOS
brew), the ready-made HF int4 container plus the self-convert path, and an
honest 'what to expect' on disk-bound speed. Commands verified against
setup.sh and the coli subcommands; every cross-linked doc exists. Linked
from the README's Get started section.

Closes #414

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 14:08:19 +02:00
Vincenzo Fornaro 3ffe4bb75e Merge pull request #413 from JustVugg/p-sec-trustboundary
security: reject malformed model tensors at the untrusted-mirror boundary (C half of #368)
2026-07-19 13:11:06 +02:00
3 changed files with 173 additions and 0 deletions
+4
View File
@@ -151,6 +151,10 @@ scale-granularity/rotation ablations live in
## Get started
> **New here?** The [Quick Start guide](docs/quickstart.md) walks through
> install → build → model → first chat step by step for Linux, Windows, and
> macOS, with copy-paste commands and no assumed background.
### 1. Get the model
A pre-converted **GLM-5.2 int4** container is on Hugging Face — **use the
+5
View File
@@ -2931,6 +2931,11 @@ static void moe(Model *m, Layer *l, int layer, float *x, int S, float *out, int
m->eusage[layer][idxs[(int64_t)s*K+kk]]++;
ehit_mark(m,layer,idxs[(int64_t)s*K+kk]);
if(m->eheat[layer][idxs[(int64_t)s*K+kk]]<UINT32_MAX) m->eheat[layer][idxs[(int64_t)s*K+kk]]++;
/* #417: la scorciatoia GPU-prerouted deve far avanzare l'orologio di recency
* come il percorso router completo (riga ~3055), altrimenti elast/eaccess_clock
* si congelano a fine prefill e il tie-breaker LFRU di REPIN gira su punteggi
* stantii durante il decode su Metal. */
m->elast[layer][idxs[(int64_t)s*K+kk]]=++m->eaccess_clock;
}
for(int d=0;d<D;d++) out[(int64_t)s*D+d]=0;
}
+164
View File
@@ -0,0 +1,164 @@
# Quick Start — from zero to a running model
A step-by-step guide for first-time users on **Linux**, **Windows**, and **macOS**.
No prior experience with C, CUDA, or model conversion is assumed. If you get
stuck, `./coli doctor` (below) tells you exactly what's missing.
> **What you're setting up:** colibrì runs a very large Mixture-of-Experts model
> (e.g. GLM-5.2, 744B parameters) on a normal machine by streaming the model's
> experts from disk instead of needing them all in RAM. The engine is a single
> C program; Python is only used once, to prepare the model files.
---
## 0. What you need first (prerequisites)
| | Minimum | Recommended |
|---|---|---|
| **RAM** | ~16 GB | 24 GB+ |
| **Free disk** | ~380 GB for the int4 model | a fast NVMe SSD (streaming speed = your token speed) |
| **OS** | Linux, Windows 10/11, or macOS | any |
| **Tools** | a C compiler + `make` + `git` + `python3` | — |
You do **not** need a GPU. A GPU only helps if you have one; the engine runs
CPU-only by default.
---
## 1. Install the build tools
### Linux (Ubuntu / Debian)
```bash
sudo apt update
sudo apt install -y build-essential git python3
```
`build-essential` gives you `gcc`, `make`, and OpenMP (libgomp) — everything the
engine needs.
### Windows
You have two options.
**Option A — download a prebuilt binary (no compiler needed).**
Grab the latest `colibri-<version>-windows-x86_64.zip` from the
[Releases page](https://github.com/JustVugg/colibri/releases), unzip it, and
skip to [step 3](#3-get-the-model). Python 3 (from
[python.org](https://www.python.org/downloads/)) is still needed if you want to
convert a model yourself.
**Option B — build from source with MSYS2.**
Install [MSYS2](https://www.msys2.org/), open the **UCRT64** shell, and run:
```bash
pacman -S --needed mingw-w64-ucrt-x86_64-gcc make git python
```
### macOS
```bash
xcode-select --install # C compiler (clang)
brew install libomp git python # OpenMP for multithreading
```
---
## 2. Get the code and build the engine
```bash
git clone https://github.com/JustVugg/colibri.git
cd colibri/c
./setup.sh
```
`setup.sh` checks your compiler and OpenMP, builds the engine, and runs a tiny
self-test. When it prints:
```
engine self-test: 32/32 (expected 32/32)
```
the engine is working correctly. (On Windows Option A you already have the
binary — you can skip this step.)
---
## 3. Get the model
You have two paths.
### Easiest — download a ready-made int4 container
A pre-converted **GLM-5.2 int4** model is on Hugging Face. **Use the version
with the int8 MTP heads** (the plain int4 heads disable speculative decoding —
see [#8](https://github.com/JustVugg/colibri/issues/8)):
**https://huggingface.co/mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp**
Download it into a folder on a fast disk, e.g. `/nvme/glm52_i4` (Linux/macOS) or
`D:\glm52_i4` (Windows). It is about **372 GB**, so make sure you have the space.
### Or convert it yourself from the FP8 source
One resumable command downloads and converts the model shard by shard, so it
never needs the full ~756 GB on disk at once:
```bash
./coli convert --model /nvme/glm52_i4
```
This step uses Python and runs only once. Safe to interrupt and re-run — it
resumes where it left off.
---
## 4. Run it
Point `COLI_MODEL` at the folder from step 3 and start chatting:
```bash
# Linux / macOS
COLI_MODEL=/nvme/glm52_i4 ./coli chat
# Windows (UCRT64 shell)
COLI_MODEL=/d/glm52_i4 ./coli chat
```
Useful first commands:
```bash
COLI_MODEL=/nvme/glm52_i4 ./coli doctor # read-only check: is everything ready?
COLI_MODEL=/nvme/glm52_i4 ./coli plan # shows where the model will live (RAM/disk/GPU)
COLI_MODEL=/nvme/glm52_i4 ./coli chat --topp 0.85 # faster: reads less from disk, same quality
```
> **Tip:** `--topp 0.85` is worth adding on a disk-bound machine — it reads
> fewer expert bytes per token with no quality loss, which directly means more
> tokens per second.
---
## 5. What to expect
- **First launch loads the resident weights** (~10 GB) — this takes a moment.
- **Speed depends on your disk.** The experts stream from storage, so a fast
NVMe SSD is the single biggest factor in tokens/second. On a slow or shared
disk, generation can be well under 1 token/second — that's expected, and it's
the honest cost of running a 744B model on a small machine.
- **It's still the full model.** Placement only changes speed, never the model's
answers or precision.
If something doesn't work, run `./coli doctor` — it reports exactly what's
missing (compiler, model files, permissions) and how to fix it.
---
## Where to go next
| Topic | Doc |
|---|---|
| Windows native build (and CUDA DLL) | [docs/windows.md](windows.md) |
| Tuning: cache, prefetch, speculation | [docs/tuning.md](tuning.md) |
| OpenAI-compatible API + web dashboard | [docs/api.md](api.md) |
| Every environment variable | [docs/ENVIRONMENT.md](ENVIRONMENT.md) |