Compare commits
3 Commits
dev
...
site-to-main
| Author | SHA1 | Date | |
|---|---|---|---|
| c6801451bd | |||
| ff93ae2bc1 | |||
| ec087f6bca |
@@ -0,0 +1,36 @@
|
|||||||
|
name: Deploy website
|
||||||
|
|
||||||
|
# Publishes site/ to GitHub Pages. One-time repo setup:
|
||||||
|
# Settings → Pages → Build and deployment → Source: "GitHub Actions".
|
||||||
|
# Custom domain later: add site/CNAME with the bare domain, point DNS
|
||||||
|
# (A/AAAA to GitHub Pages IPs or CNAME to <org>.github.io), done.
|
||||||
|
|
||||||
|
on:
|
||||||
|
push:
|
||||||
|
branches: [main]
|
||||||
|
paths: ['site/**', '.github/workflows/site.yml']
|
||||||
|
workflow_dispatch:
|
||||||
|
|
||||||
|
permissions:
|
||||||
|
contents: read
|
||||||
|
pages: write
|
||||||
|
id-token: write
|
||||||
|
|
||||||
|
concurrency:
|
||||||
|
group: pages
|
||||||
|
cancel-in-progress: true
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
deploy:
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
environment:
|
||||||
|
name: github-pages
|
||||||
|
url: ${{ steps.deployment.outputs.page_url }}
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v4
|
||||||
|
- uses: actions/configure-pages@v5
|
||||||
|
- uses: actions/upload-pages-artifact@v3
|
||||||
|
with:
|
||||||
|
path: site
|
||||||
|
- id: deployment
|
||||||
|
uses: actions/deploy-pages@v4
|
||||||
@@ -151,6 +151,15 @@ scale-granularity/rotation ablations live in
|
|||||||
|
|
||||||
## Get started
|
## Get started
|
||||||
|
|
||||||
|
> **New here, or on Windows?** The [Quick Start guide](docs/quickstart.md) walks
|
||||||
|
> through install → build → model → first chat step by step for Linux, Windows,
|
||||||
|
> and macOS. On **Windows** you don't even need to build: download the
|
||||||
|
> `colibri-<version>-windows-x86_64.zip` from
|
||||||
|
> [Releases](https://github.com/JustVugg/colibri/releases), unzip it, rename
|
||||||
|
> `colibri-*-windows-x86_64.exe` → `glm.exe`, install
|
||||||
|
> [Python 3](https://www.python.org/downloads/), and run `coli chat` — full
|
||||||
|
> details in the [Windows section](docs/quickstart.md#windows).
|
||||||
|
|
||||||
### 1. Get the model
|
### 1. Get the model
|
||||||
|
|
||||||
A pre-converted **GLM-5.2 int4** container is on Hugging Face — **use the
|
A pre-converted **GLM-5.2 int4** container is on Hugging Face — **use the
|
||||||
|
|||||||
@@ -0,0 +1,184 @@
|
|||||||
|
# Quick Start — from zero to a running model
|
||||||
|
|
||||||
|
A step-by-step guide for first-time users on **Linux**, **Windows**, and **macOS**.
|
||||||
|
No prior experience with C, CUDA, or model conversion is assumed. If you get
|
||||||
|
stuck, `./coli doctor` (below) tells you exactly what's missing.
|
||||||
|
|
||||||
|
> **What you're setting up:** colibrì runs a very large Mixture-of-Experts model
|
||||||
|
> (e.g. GLM-5.2, 744B parameters) on a normal machine by streaming the model's
|
||||||
|
> experts from disk instead of needing them all in RAM. The engine is a single
|
||||||
|
> C program; Python is only used once, to prepare the model files.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 0. What you need first (prerequisites)
|
||||||
|
|
||||||
|
| | Minimum | Recommended |
|
||||||
|
|---|---|---|
|
||||||
|
| **RAM** | ~16 GB | 24 GB+ |
|
||||||
|
| **Free disk** | ~380 GB for the int4 model | a fast NVMe SSD (streaming speed = your token speed) |
|
||||||
|
| **OS** | Linux, Windows 10/11, or macOS | any |
|
||||||
|
| **Tools** | a C compiler + `make` + `git` + `python3` | — |
|
||||||
|
|
||||||
|
You do **not** need a GPU. A GPU only helps if you have one; the engine runs
|
||||||
|
CPU-only by default.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Install the build tools
|
||||||
|
|
||||||
|
### Linux (Ubuntu / Debian)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo apt update
|
||||||
|
sudo apt install -y build-essential git python3
|
||||||
|
```
|
||||||
|
|
||||||
|
`build-essential` gives you `gcc`, `make`, and OpenMP (libgomp) — everything the
|
||||||
|
engine needs.
|
||||||
|
|
||||||
|
### Windows
|
||||||
|
|
||||||
|
You have two options.
|
||||||
|
|
||||||
|
**Option A — download a prebuilt binary (no compiler needed).**
|
||||||
|
Grab `colibri-<version>-windows-x86_64.zip` from the
|
||||||
|
[Releases page](https://github.com/JustVugg/colibri/releases) and unzip it.
|
||||||
|
Inside you'll find:
|
||||||
|
|
||||||
|
| File | What it is |
|
||||||
|
|---|---|
|
||||||
|
| `colibri-<version>-windows-x86_64.exe` | **the engine** — the C program that actually runs the model |
|
||||||
|
| `coli` | the command-line launcher (`chat`, `serve`, `convert`, `doctor`, …) |
|
||||||
|
| `openai_server.py`, `resource_plan.py`, `doctor.py` | Python support for the API server and placement planner |
|
||||||
|
|
||||||
|
Two setup steps:
|
||||||
|
|
||||||
|
1. **Rename the engine to `glm.exe`** so the launcher can find it (it looks for a
|
||||||
|
binary named `glm`):
|
||||||
|
```powershell
|
||||||
|
Rename-Item colibri-*-windows-x86_64.exe glm.exe
|
||||||
|
```
|
||||||
|
2. **Install Python 3** from [python.org](https://www.python.org/downloads/) — the
|
||||||
|
`coli` launcher and the API gateway are Python scripts (the engine itself is
|
||||||
|
pure C and needs nothing).
|
||||||
|
|
||||||
|
Then continue to [step 3](#3-get-the-model). Prefer to skip the launcher? You can
|
||||||
|
run the engine directly — `.\glm.exe` reads the model path from the `SNAP`
|
||||||
|
environment variable (see [docs/windows.md](windows.md)) — but `coli chat` is the
|
||||||
|
easy path.
|
||||||
|
|
||||||
|
**Option B — build from source with MSYS2.**
|
||||||
|
Install [MSYS2](https://www.msys2.org/), open the **UCRT64** shell, and run:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pacman -S --needed mingw-w64-ucrt-x86_64-gcc make git python
|
||||||
|
```
|
||||||
|
|
||||||
|
### macOS
|
||||||
|
|
||||||
|
```bash
|
||||||
|
xcode-select --install # C compiler (clang)
|
||||||
|
brew install libomp git python # OpenMP for multithreading
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Get the code and build the engine
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git clone https://github.com/JustVugg/colibri.git
|
||||||
|
cd colibri/c
|
||||||
|
./setup.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
`setup.sh` checks your compiler and OpenMP, builds the engine, and runs a tiny
|
||||||
|
self-test. When it prints:
|
||||||
|
|
||||||
|
```
|
||||||
|
engine self-test: 32/32 (expected 32/32)
|
||||||
|
```
|
||||||
|
|
||||||
|
the engine is working correctly. (On Windows Option A you already have the
|
||||||
|
binary — you can skip this step.)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Get the model
|
||||||
|
|
||||||
|
You have two paths.
|
||||||
|
|
||||||
|
### Easiest — download a ready-made int4 container
|
||||||
|
|
||||||
|
A pre-converted **GLM-5.2 int4** model is on Hugging Face. **Use the version
|
||||||
|
with the int8 MTP heads** (the plain int4 heads disable speculative decoding —
|
||||||
|
see [#8](https://github.com/JustVugg/colibri/issues/8)):
|
||||||
|
|
||||||
|
**https://huggingface.co/mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp**
|
||||||
|
|
||||||
|
Download it into a folder on a fast disk, e.g. `/nvme/glm52_i4` (Linux/macOS) or
|
||||||
|
`D:\glm52_i4` (Windows). It is about **372 GB**, so make sure you have the space.
|
||||||
|
|
||||||
|
### Or convert it yourself from the FP8 source
|
||||||
|
|
||||||
|
One resumable command downloads and converts the model shard by shard, so it
|
||||||
|
never needs the full ~756 GB on disk at once:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./coli convert --model /nvme/glm52_i4
|
||||||
|
```
|
||||||
|
|
||||||
|
This step uses Python and runs only once. Safe to interrupt and re-run — it
|
||||||
|
resumes where it left off.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Run it
|
||||||
|
|
||||||
|
Point `COLI_MODEL` at the folder from step 3 and start chatting:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Linux / macOS
|
||||||
|
COLI_MODEL=/nvme/glm52_i4 ./coli chat
|
||||||
|
|
||||||
|
# Windows (UCRT64 shell)
|
||||||
|
COLI_MODEL=/d/glm52_i4 ./coli chat
|
||||||
|
```
|
||||||
|
|
||||||
|
Useful first commands:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
COLI_MODEL=/nvme/glm52_i4 ./coli doctor # read-only check: is everything ready?
|
||||||
|
COLI_MODEL=/nvme/glm52_i4 ./coli plan # shows where the model will live (RAM/disk/GPU)
|
||||||
|
COLI_MODEL=/nvme/glm52_i4 ./coli chat --topp 0.85 # faster: reads less from disk, same quality
|
||||||
|
```
|
||||||
|
|
||||||
|
> **Tip:** `--topp 0.85` is worth adding on a disk-bound machine — it reads
|
||||||
|
> fewer expert bytes per token with no quality loss, which directly means more
|
||||||
|
> tokens per second.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. What to expect
|
||||||
|
|
||||||
|
- **First launch loads the resident weights** (~10 GB) — this takes a moment.
|
||||||
|
- **Speed depends on your disk.** The experts stream from storage, so a fast
|
||||||
|
NVMe SSD is the single biggest factor in tokens/second. On a slow or shared
|
||||||
|
disk, generation can be well under 1 token/second — that's expected, and it's
|
||||||
|
the honest cost of running a 744B model on a small machine.
|
||||||
|
- **It's still the full model.** Placement only changes speed, never the model's
|
||||||
|
answers or precision.
|
||||||
|
|
||||||
|
If something doesn't work, run `./coli doctor` — it reports exactly what's
|
||||||
|
missing (compiler, model files, permissions) and how to fix it.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Where to go next
|
||||||
|
|
||||||
|
| Topic | Doc |
|
||||||
|
|---|---|
|
||||||
|
| Windows native build (and CUDA DLL) | [docs/windows.md](windows.md) |
|
||||||
|
| Tuning: cache, prefetch, speculation | [docs/tuning.md](tuning.md) |
|
||||||
|
| OpenAI-compatible API + web dashboard | [docs/api.md](api.md) |
|
||||||
|
| Every environment variable | [docs/ENVIRONMENT.md](ENVIRONMENT.md) |
|
||||||
@@ -0,0 +1,50 @@
|
|||||||
|
<svg xmlns="http://www.w3.org/2000/svg" width="512" height="512" viewBox="0 0 168 168">
|
||||||
|
<rect width="168" height="168" rx="36" fill="#080b0d"/>
|
||||||
|
<g transform="translate(7 21)" shape-rendering="crispEdges">
|
||||||
|
<rect x="56" y="0" width="14" height="14" fill="#d75fd7"/>
|
||||||
|
<rect x="70" y="0" width="14" height="14" fill="#d75fd7"/>
|
||||||
|
<rect x="84" y="0" width="14" height="14" fill="#d75fd7"/>
|
||||||
|
<rect x="42" y="14" width="14" height="14" fill="#d75fd7"/>
|
||||||
|
<rect x="56" y="14" width="14" height="14" fill="#d75fd7"/>
|
||||||
|
<rect x="70" y="14" width="14" height="14" fill="#d75fd7"/>
|
||||||
|
<rect x="84" y="14" width="14" height="14" fill="#d75fd7"/>
|
||||||
|
<rect x="98" y="14" width="14" height="14" fill="#d75fd7"/>
|
||||||
|
<rect x="140" y="14" width="14" height="14" fill="#5fd7d7"/>
|
||||||
|
<rect x="56" y="28" width="14" height="14" fill="#d75fd7"/>
|
||||||
|
<rect x="70" y="28" width="14" height="14" fill="#d75fd7"/>
|
||||||
|
<rect x="84" y="28" width="14" height="14" fill="#d75fd7"/>
|
||||||
|
<rect x="98" y="28" width="14" height="14" fill="#d75fd7"/>
|
||||||
|
<rect x="126" y="28" width="14" height="14" fill="#5fd7d7"/>
|
||||||
|
<rect x="140" y="28" width="14" height="14" fill="#5fd7d7"/>
|
||||||
|
<rect x="0" y="42" width="14" height="14" fill="#ff8700"/>
|
||||||
|
<rect x="14" y="42" width="14" height="14" fill="#ff8700"/>
|
||||||
|
<rect x="28" y="42" width="14" height="14" fill="#ff8700"/>
|
||||||
|
<rect x="42" y="42" width="14" height="14" fill="#ff8700"/>
|
||||||
|
<rect x="56" y="42" width="14" height="14" fill="#00afaf"/>
|
||||||
|
<rect x="70" y="42" width="14" height="14" fill="#00afaf"/>
|
||||||
|
<rect x="84" y="42" width="14" height="14" fill="#fff"/>
|
||||||
|
<rect x="98" y="42" width="14" height="14" fill="#00afaf"/>
|
||||||
|
<rect x="112" y="42" width="14" height="14" fill="#5fd7d7"/>
|
||||||
|
<rect x="126" y="42" width="14" height="14" fill="#5fd7d7"/>
|
||||||
|
<rect x="56" y="56" width="14" height="14" fill="#00afaf"/>
|
||||||
|
<rect x="70" y="56" width="14" height="14" fill="#00afaf"/>
|
||||||
|
<rect x="84" y="56" width="14" height="14" fill="#00afaf"/>
|
||||||
|
<rect x="98" y="56" width="14" height="14" fill="#00afaf"/>
|
||||||
|
<rect x="112" y="56" width="14" height="14" fill="#00afaf"/>
|
||||||
|
<rect x="126" y="56" width="14" height="14" fill="#5fd7d7"/>
|
||||||
|
<rect x="140" y="56" width="14" height="14" fill="#5fd7d7"/>
|
||||||
|
<rect x="70" y="70" width="14" height="14" fill="#00afaf"/>
|
||||||
|
<rect x="84" y="70" width="14" height="14" fill="#00afaf"/>
|
||||||
|
<rect x="98" y="70" width="14" height="14" fill="#00afaf"/>
|
||||||
|
<rect x="112" y="70" width="14" height="14" fill="#00afaf"/>
|
||||||
|
<rect x="126" y="70" width="14" height="14" fill="#5fd7d7"/>
|
||||||
|
<rect x="140" y="70" width="14" height="14" fill="#5fd7d7"/>
|
||||||
|
<rect x="84" y="84" width="14" height="14" fill="#00afaf"/>
|
||||||
|
<rect x="98" y="84" width="14" height="14" fill="#00afaf"/>
|
||||||
|
<rect x="112" y="84" width="14" height="14" fill="#5fd7d7"/>
|
||||||
|
<rect x="126" y="84" width="14" height="14" fill="#5fd7d7"/>
|
||||||
|
<rect x="98" y="98" width="14" height="14" fill="#00afaf"/>
|
||||||
|
<rect x="112" y="98" width="14" height="14" fill="#5fd7d7"/>
|
||||||
|
<rect x="112" y="112" width="14" height="14" fill="#5fd7d7"/>
|
||||||
|
</g>
|
||||||
|
</svg>
|
||||||
|
After Width: | Height: | Size: 3.0 KiB |
@@ -0,0 +1,55 @@
|
|||||||
|
<svg xmlns="http://www.w3.org/2000/svg" width="620" height="140" viewBox="0 0 620 140">
|
||||||
|
<g shape-rendering="crispEdges">
|
||||||
|
<rect x="56" y="0" width="14" height="14" fill="#d75fd7"/>
|
||||||
|
<rect x="70" y="0" width="14" height="14" fill="#d75fd7"/>
|
||||||
|
<rect x="84" y="0" width="14" height="14" fill="#d75fd7"/>
|
||||||
|
<rect x="42" y="14" width="14" height="14" fill="#d75fd7"/>
|
||||||
|
<rect x="56" y="14" width="14" height="14" fill="#d75fd7"/>
|
||||||
|
<rect x="70" y="14" width="14" height="14" fill="#d75fd7"/>
|
||||||
|
<rect x="84" y="14" width="14" height="14" fill="#d75fd7"/>
|
||||||
|
<rect x="98" y="14" width="14" height="14" fill="#d75fd7"/>
|
||||||
|
<rect x="140" y="14" width="14" height="14" fill="#5fd7d7"/>
|
||||||
|
<rect x="56" y="28" width="14" height="14" fill="#d75fd7"/>
|
||||||
|
<rect x="70" y="28" width="14" height="14" fill="#d75fd7"/>
|
||||||
|
<rect x="84" y="28" width="14" height="14" fill="#d75fd7"/>
|
||||||
|
<rect x="98" y="28" width="14" height="14" fill="#d75fd7"/>
|
||||||
|
<rect x="126" y="28" width="14" height="14" fill="#5fd7d7"/>
|
||||||
|
<rect x="140" y="28" width="14" height="14" fill="#5fd7d7"/>
|
||||||
|
<rect x="0" y="42" width="14" height="14" fill="#ff8700"/>
|
||||||
|
<rect x="14" y="42" width="14" height="14" fill="#ff8700"/>
|
||||||
|
<rect x="28" y="42" width="14" height="14" fill="#ff8700"/>
|
||||||
|
<rect x="42" y="42" width="14" height="14" fill="#ff8700"/>
|
||||||
|
<rect x="56" y="42" width="14" height="14" fill="#00afaf"/>
|
||||||
|
<rect x="70" y="42" width="14" height="14" fill="#00afaf"/>
|
||||||
|
<rect x="84" y="42" width="14" height="14" fill="#ffffff"/>
|
||||||
|
<rect x="98" y="42" width="14" height="14" fill="#00afaf"/>
|
||||||
|
<rect x="112" y="42" width="14" height="14" fill="#5fd7d7"/>
|
||||||
|
<rect x="126" y="42" width="14" height="14" fill="#5fd7d7"/>
|
||||||
|
<rect x="56" y="56" width="14" height="14" fill="#00afaf"/>
|
||||||
|
<rect x="70" y="56" width="14" height="14" fill="#00afaf"/>
|
||||||
|
<rect x="84" y="56" width="14" height="14" fill="#00afaf"/>
|
||||||
|
<rect x="98" y="56" width="14" height="14" fill="#00afaf"/>
|
||||||
|
<rect x="112" y="56" width="14" height="14" fill="#00afaf"/>
|
||||||
|
<rect x="126" y="56" width="14" height="14" fill="#5fd7d7"/>
|
||||||
|
<rect x="140" y="56" width="14" height="14" fill="#5fd7d7"/>
|
||||||
|
<rect x="70" y="70" width="14" height="14" fill="#00afaf"/>
|
||||||
|
<rect x="84" y="70" width="14" height="14" fill="#00afaf"/>
|
||||||
|
<rect x="98" y="70" width="14" height="14" fill="#00afaf"/>
|
||||||
|
<rect x="112" y="70" width="14" height="14" fill="#00afaf"/>
|
||||||
|
<rect x="126" y="70" width="14" height="14" fill="#5fd7d7"/>
|
||||||
|
<rect x="140" y="70" width="14" height="14" fill="#5fd7d7"/>
|
||||||
|
<rect x="84" y="84" width="14" height="14" fill="#00afaf"/>
|
||||||
|
<rect x="98" y="84" width="14" height="14" fill="#00afaf"/>
|
||||||
|
<rect x="112" y="84" width="14" height="14" fill="#5fd7d7"/>
|
||||||
|
<rect x="126" y="84" width="14" height="14" fill="#5fd7d7"/>
|
||||||
|
<rect x="98" y="98" width="14" height="14" fill="#00afaf"/>
|
||||||
|
<rect x="112" y="98" width="14" height="14" fill="#5fd7d7"/>
|
||||||
|
<rect x="112" y="112" width="14" height="14" fill="#5fd7d7"/>
|
||||||
|
</g>
|
||||||
|
<text x="252" y="62" font-family="ui-monospace, SFMono-Regular, Menlo, Consolas, monospace"
|
||||||
|
font-size="52" font-weight="bold" fill="#00afaf">colibrì</text>
|
||||||
|
<text x="252" y="94" font-family="ui-monospace, SFMono-Regular, Menlo, Consolas, monospace"
|
||||||
|
font-size="19" fill="#808080" font-style="italic">tiny engine, immense model</text>
|
||||||
|
<text x="252" y="122" font-family="ui-monospace, SFMono-Regular, Menlo, Consolas, monospace"
|
||||||
|
font-size="15" fill="#9a9a9a">GLM-5.2 · 744B MoE · int4 · streaming CPU</text>
|
||||||
|
</svg>
|
||||||
|
After Width: | Height: | Size: 3.5 KiB |
+698
File diff suppressed because one or more lines are too long
Reference in New Issue
Block a user