diff --git a/README.md b/README.md index 797f1ca..085e4e9 100644 --- a/README.md +++ b/README.md @@ -151,6 +151,15 @@ scale-granularity/rotation ablations live in ## Get started +> **New here, or on Windows?** The [Quick Start guide](docs/quickstart.md) walks +> through install → build → model → first chat step by step for Linux, Windows, +> and macOS. On **Windows** you don't even need to build: download the +> `colibri--windows-x86_64.zip` from +> [Releases](https://github.com/JustVugg/colibri/releases), unzip it, rename +> `colibri-*-windows-x86_64.exe` → `glm.exe`, install +> [Python 3](https://www.python.org/downloads/), and run `coli chat` — full +> details in the [Windows section](docs/quickstart.md#windows). + ### 1. Get the model A pre-converted **GLM-5.2 int4** container is on Hugging Face — **use the diff --git a/docs/quickstart.md b/docs/quickstart.md new file mode 100644 index 0000000..60d4877 --- /dev/null +++ b/docs/quickstart.md @@ -0,0 +1,184 @@ +# Quick Start — from zero to a running model + +A step-by-step guide for first-time users on **Linux**, **Windows**, and **macOS**. +No prior experience with C, CUDA, or model conversion is assumed. If you get +stuck, `./coli doctor` (below) tells you exactly what's missing. + +> **What you're setting up:** colibrì runs a very large Mixture-of-Experts model +> (e.g. GLM-5.2, 744B parameters) on a normal machine by streaming the model's +> experts from disk instead of needing them all in RAM. The engine is a single +> C program; Python is only used once, to prepare the model files. + +--- + +## 0. What you need first (prerequisites) + +| | Minimum | Recommended | +|---|---|---| +| **RAM** | ~16 GB | 24 GB+ | +| **Free disk** | ~380 GB for the int4 model | a fast NVMe SSD (streaming speed = your token speed) | +| **OS** | Linux, Windows 10/11, or macOS | any | +| **Tools** | a C compiler + `make` + `git` + `python3` | — | + +You do **not** need a GPU. A GPU only helps if you have one; the engine runs +CPU-only by default. + +--- + +## 1. Install the build tools + +### Linux (Ubuntu / Debian) + +```bash +sudo apt update +sudo apt install -y build-essential git python3 +``` + +`build-essential` gives you `gcc`, `make`, and OpenMP (libgomp) — everything the +engine needs. + +### Windows + +You have two options. + +**Option A — download a prebuilt binary (no compiler needed).** +Grab `colibri--windows-x86_64.zip` from the +[Releases page](https://github.com/JustVugg/colibri/releases) and unzip it. +Inside you'll find: + +| File | What it is | +|---|---| +| `colibri--windows-x86_64.exe` | **the engine** — the C program that actually runs the model | +| `coli` | the command-line launcher (`chat`, `serve`, `convert`, `doctor`, …) | +| `openai_server.py`, `resource_plan.py`, `doctor.py` | Python support for the API server and placement planner | + +Two setup steps: + +1. **Rename the engine to `glm.exe`** so the launcher can find it (it looks for a + binary named `glm`): + ```powershell + Rename-Item colibri-*-windows-x86_64.exe glm.exe + ``` +2. **Install Python 3** from [python.org](https://www.python.org/downloads/) — the + `coli` launcher and the API gateway are Python scripts (the engine itself is + pure C and needs nothing). + +Then continue to [step 3](#3-get-the-model). Prefer to skip the launcher? You can +run the engine directly — `.\glm.exe` reads the model path from the `SNAP` +environment variable (see [docs/windows.md](windows.md)) — but `coli chat` is the +easy path. + +**Option B — build from source with MSYS2.** +Install [MSYS2](https://www.msys2.org/), open the **UCRT64** shell, and run: + +```bash +pacman -S --needed mingw-w64-ucrt-x86_64-gcc make git python +``` + +### macOS + +```bash +xcode-select --install # C compiler (clang) +brew install libomp git python # OpenMP for multithreading +``` + +--- + +## 2. Get the code and build the engine + +```bash +git clone https://github.com/JustVugg/colibri.git +cd colibri/c +./setup.sh +``` + +`setup.sh` checks your compiler and OpenMP, builds the engine, and runs a tiny +self-test. When it prints: + +``` +engine self-test: 32/32 (expected 32/32) +``` + +the engine is working correctly. (On Windows Option A you already have the +binary — you can skip this step.) + +--- + +## 3. Get the model + +You have two paths. + +### Easiest — download a ready-made int4 container + +A pre-converted **GLM-5.2 int4** model is on Hugging Face. **Use the version +with the int8 MTP heads** (the plain int4 heads disable speculative decoding — +see [#8](https://github.com/JustVugg/colibri/issues/8)): + +**https://huggingface.co/mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp** + +Download it into a folder on a fast disk, e.g. `/nvme/glm52_i4` (Linux/macOS) or +`D:\glm52_i4` (Windows). It is about **372 GB**, so make sure you have the space. + +### Or convert it yourself from the FP8 source + +One resumable command downloads and converts the model shard by shard, so it +never needs the full ~756 GB on disk at once: + +```bash +./coli convert --model /nvme/glm52_i4 +``` + +This step uses Python and runs only once. Safe to interrupt and re-run — it +resumes where it left off. + +--- + +## 4. Run it + +Point `COLI_MODEL` at the folder from step 3 and start chatting: + +```bash +# Linux / macOS +COLI_MODEL=/nvme/glm52_i4 ./coli chat + +# Windows (UCRT64 shell) +COLI_MODEL=/d/glm52_i4 ./coli chat +``` + +Useful first commands: + +```bash +COLI_MODEL=/nvme/glm52_i4 ./coli doctor # read-only check: is everything ready? +COLI_MODEL=/nvme/glm52_i4 ./coli plan # shows where the model will live (RAM/disk/GPU) +COLI_MODEL=/nvme/glm52_i4 ./coli chat --topp 0.85 # faster: reads less from disk, same quality +``` + +> **Tip:** `--topp 0.85` is worth adding on a disk-bound machine — it reads +> fewer expert bytes per token with no quality loss, which directly means more +> tokens per second. + +--- + +## 5. What to expect + +- **First launch loads the resident weights** (~10 GB) — this takes a moment. +- **Speed depends on your disk.** The experts stream from storage, so a fast + NVMe SSD is the single biggest factor in tokens/second. On a slow or shared + disk, generation can be well under 1 token/second — that's expected, and it's + the honest cost of running a 744B model on a small machine. +- **It's still the full model.** Placement only changes speed, never the model's + answers or precision. + +If something doesn't work, run `./coli doctor` — it reports exactly what's +missing (compiler, model files, permissions) and how to fix it. + +--- + +## Where to go next + +| Topic | Doc | +|---|---| +| Windows native build (and CUDA DLL) | [docs/windows.md](windows.md) | +| Tuning: cache, prefetch, speculation | [docs/tuning.md](tuning.md) | +| OpenAI-compatible API + web dashboard | [docs/api.md](api.md) | +| Every environment variable | [docs/ENVIRONMENT.md](ENVIRONMENT.md) |