diff --git a/.github/workflows/site.yml b/.github/workflows/site.yml new file mode 100644 index 0000000..160cd61 --- /dev/null +++ b/.github/workflows/site.yml @@ -0,0 +1,36 @@ +name: Deploy website + +# Publishes site/ to GitHub Pages. One-time repo setup: +# Settings → Pages → Build and deployment → Source: "GitHub Actions". +# Custom domain later: add site/CNAME with the bare domain, point DNS +# (A/AAAA to GitHub Pages IPs or CNAME to .github.io), done. + +on: + push: + branches: [main] + paths: ['site/**', '.github/workflows/site.yml'] + workflow_dispatch: + +permissions: + contents: read + pages: write + id-token: write + +concurrency: + group: pages + cancel-in-progress: true + +jobs: + deploy: + runs-on: ubuntu-latest + environment: + name: github-pages + url: ${{ steps.deployment.outputs.page_url }} + steps: + - uses: actions/checkout@v4 + - uses: actions/configure-pages@v5 + - uses: actions/upload-pages-artifact@v3 + with: + path: site + - id: deployment + uses: actions/deploy-pages@v4 diff --git a/README.md b/README.md index 48d0209..152274f 100644 --- a/README.md +++ b/README.md @@ -44,6 +44,16 @@ brightness is routing heat, and every expert routed in a turn flashes white. Hov as a 3-D galaxy — 13,260 characterised experts, 1,041 replicated specialists clustering by topic (poetry, law, Chinese, SQL…). Position is measured routing affinity, not a learned embedding. Drag to spin.

+## The vision + +Frontier models should not be sealed inside datacenters. colibrì exists so that +**anyone curious enough can open one up**: run a 744B-parameter mind on hardware +you already own, watch every expert fire in real time, and change the code that +does it. Not renting intelligence behind an API — *holding* it: probing it, +measuring it, improving it. Every optimisation in this project started with +someone measuring something on their own machine; the engine is deliberately +small enough that the next one can come from you. + ## The idea A 744B Mixture-of-Experts model activates only ~40B parameters per token — and @@ -61,6 +71,18 @@ So the model doesn't need to *fit* in fast memory — it needs to be **placed**: at int4) live **on disk** (~370 GB) and are **streamed on demand**, with a per-layer LRU cache, a learned pinned hot-store, and an optional VRAM tier. +Think of the core algorithm as **a JIT, but for weights**. A compiler JIT never +compiles the whole program — it watches what actually runs and compiles the hot +paths, just in time. colibrì makes the same bet about a 744B parameter space: +parameters are not resident state to be held, they are **data to be staged** +across a heterogeneous storage hierarchy (VRAM / RAM / NVMe), exactly when the +router proves they are needed. Measured routing heat decides which experts earn +which tier, the router runs a layer ahead so prefetch hides the staging latency, +and — like a JIT — the engine learns your workload: the more you run, the hotter +the right experts get. It works because routing has measurable structure (see +the [expert atlas](https://github.com/JustVugg/colibri/issues/175)) — and +structure is cacheable. + The engine is a single C file (`c/glm.c`) plus small headers. No BLAS, no Python at runtime, no GPU required. @@ -211,6 +233,18 @@ install from the clone, not a standalone wheel). | Grammar-forced drafts (structured output) | [docs/grammar-draft.md](docs/grammar-draft.md) | | Environment variable inventory | [docs/ENVIRONMENT.md](docs/ENVIRONMENT.md) | +## What's next + +- **Algorithmic research is active.** The current hierarchy is LRU + a learned + pin set; the next step is under way — smarter placement and scheduling, + overlap of CPU and GPU expert execution, and routing-aware speculation. + Everything lands the way this project always works: measured, reviewed, and + merged in the open. +- **More open models.** The tiering algorithm is model-agnostic: any MoE with + routed experts can be staged the same way. GLM-5.2 and OLMoE run today; + support for more open-weight families — **Kimi K2** (Moonshot AI), + **Qwen3 MoE** (Alibaba), **MiniMax** — is on the roadmap. + ## Supporting the project colibrì started as a one-person project on a 12-core laptop with 25 GB of RAM; @@ -251,6 +285,14 @@ The hummingbird weighs a few grams, hovers in place, and visits a thousand flowers a day. This engine keeps a 744-billion-parameter giant alive on hummingbird rations: 25 GB of RAM, twelve CPU cores, and a lot of disk patience. +## Acknowledgements + +colibrì is an engine; the minds it runs are a gift. Thank you to the teams +releasing frontier-class weights in the open — **Z.ai** (GLM), **Moonshot AI** +(Kimi), **Alibaba Qwen**, **MiniMax**, and **Allen AI** (OLMoE) — and to every +contributor who benchmarked, bisected, replicated an atlas run, or sent a patch. +This project is proof of what open weights make possible. + ## License Apache 2.0. GLM-5.2 weights are released by Z.ai under MIT. diff --git a/site/colibri-icon.svg b/site/colibri-icon.svg new file mode 100644 index 0000000..76838b9 --- /dev/null +++ b/site/colibri-icon.svg @@ -0,0 +1,50 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/site/colibri.svg b/site/colibri.svg new file mode 100644 index 0000000..4aed0bb --- /dev/null +++ b/site/colibri.svg @@ -0,0 +1,55 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + colibrì + tiny engine, immense model + GLM-5.2 · 744B MoE · int4 · streaming CPU + diff --git a/site/index.html b/site/index.html new file mode 100644 index 0000000..dd6dac1 --- /dev/null +++ b/site/index.html @@ -0,0 +1,698 @@ + + + + + +colibrì — tiny engine, immense model + + + + + + + + + + + + +
+ +
tiny engine · immense model
+

colibrì

+
a 744-billion-parameter mind, on your desk
+

Run GLM-5.2 — 744B Mixture-of-Experts — on hardware you already own. + Pure C, zero dependencies, experts staged just-in-time across VRAM, RAM and disk. + The galaxy behind this text is real: the model's measured expert atlas.

+
+ ★ Star on GitHub + Quick start → +
+
+
744B
parameters
+
19,456
experts
+
0
dependencies
+
25 GB
min RAM
+
9.2
tok/s measured
+
+
scroll
+
+ +
+ +
+
the vision
+

Frontier models should not be sealed inside datacenters.
+ Anyone curious enough should be able to open one up —
+ run it, watch every expert fire, and make it better.

+
+
⚡

Run it

+

A 744B frontier-class MoE answering on a 25 GB machine — the proven floor — up to + 9+ tok/s on a workstation. No cluster, no cloud bill, no API key.

+
🔬

Study it

+

The engine is a microscope: live routing telemetry, per-expert heat, and the first + published expert atlas of a 700B-class model — measured on machines like yours.

+
🛠

Improve it

+

One C file, readable in an afternoon. Every optimisation this project found came from + someone measuring something — the next one can come from you.

+
+
+ +
+
01 — the algorithm
+

A JIT, but for weights

+
+
+

A compiler JIT never compiles the whole program — it watches what actually runs + and compiles the hot paths, just in time. colibrì applies the same bet to a + 744B parameter space: a token activates only ~40B parameters, and just + ~11 GB of those change from token to token — the routed experts.

+
So parameters are not resident state to be held — they are + data to be staged, exactly when the router proves they are needed.
+

The engine keeps the dense weights resident and treats the 19,456 experts as a + heterogeneous storage hierarchy: measured routing heat decides which experts earn + VRAM, which earn pinned RAM, and which stream from NVMe. The router runs a layer + ahead, so prefetch hides the staging latency — and because routing has + measurable structure (the atlas below), the hierarchy keeps getting the answer right. + Like a JIT, it learns your workload: the more you run, the hotter the right experts get.

+

Everything is validated token-exact against the reference transformers + implementation — speed never buys drift.

+
⚗ Active research: the next algorithmic step is under way — + smarter placement and scheduling beyond LRU + pin, overlap of CPU and GPU expert execution, + and routing-aware speculation. Measured, reviewed, and merged in the open.
+
+
+
+ VRAMhottest experts · grouped GPU matmuls · + six 5090s hold the entire routed set — disk reads hit zero
+
↑ promote by measured heat ↓ demote
+
+ RAMwarm set · pinned & wired, learned from + your usage · AVX-512/NEON int4 kernels + LRU behind it
+
↑ promote ↓ demote
+
+ NVMecold tail · streamed on demand with io_uring, + prefetched a layer ahead — how 744B fits a 25 GB machine at all
+
token → router picks 8 of 256 experts × 75 layers → + gather across tiers → int4 matmul → next token
+
+
+
+ +
+
02 — the engine, live
+

Watch it think

+

A replay of the engine at work, paced to measured decode speeds from real + community hardware. The grid is all 19,456 experts (colour = storage tier, brightness = + routing heat, routed = white flash); the galaxy below it is the measured atlas — + watch different questions light different regions of the mind.

+
+
+
+
+ coli chat —
+
+
+
+
+ the brain — 75 MoE layers × 256 experts + MTP
+
+ +
+
0.0 tok/s
decode
+
—
ttft (measured)
+
—
resident hit
+
+
+ VRAM tier + RAM tier + disk tier + routed now +
+
+
+
+
+ the atlas, live — measured specialists routed by the current topic
+
+ +
routing: —
+
+
+
+

Simulation replays a fixed transcript at each profile's measured decode rate + (#82, #389, + #387, #273); + “—” means that machine's report did not publish the number — nothing here is invented. + Atlas flashes sample real measured affinity (atlas v1) for the active topic.

+
+ +
+
+
03 — the atlas
+

Measured, not assumed

+

Every point below is a real measured expert from the published + canonical atlas v1: + 721 canonical specialists confirmed across three independent kernel families + (AVX-512 VNNI · NEON · AVX2, engine-pinned, replication-gated) plus 637 gate-sensitive ones. + Position is the measured 10-topic affinity vector — a specialist sits at its topic's anchor, + a generalist drifts to the centre. Colour = top topic, large points = canonical. Drag to spin.

+
+
+ +
+
drag to spin · scroll past to release
+
+
+
+ +
+
04 — the ladder
+

Same engine, any hardware

+

Measured decode on real community machines — the hardware only changes + where the experts live. Every row links to its public report.

+ + + + + + + + + + + + + + + + + + + + +
hardwaredecodewhere the experts live
6× RTX 5090 · full residency + selective NUMA (#82)9.0–9.2 tok/s
all in VRAM+RAM · disk 0
2× Xeon Gold 6430 · 1 TB DDR5 · CPU-only (#389)5.42 tok/s
every expert pinned in RAM
DGX Spark GB10 · 121 GB unified (#161)3.33 tok/s
unified memory + CACHE_ROUTE
MacBook Pro M5 Max · 128 GB · Metal (#387)2.0 tok/s
Metal tier + warm pin
Ryzen AI Max+ 395 · 128 GB (#200)1.83 tok/s
hot set pinned in RAM
Ryzen 9 9950X3D2 · RTX 5090 · Gen5 NVMe (#120)1.23 tok/s
28 GB VRAM tier + 11.5 GB/s disk
single RTX 5070 Ti laptop-class (#273)1.07 tok/s
GPU-resident pipeline
25 GB dev box · cold — the proven floor0.05–0.1 tok/s
streamed from NVMe
+

…and 15+ more community machines, from an M4 Pro Mac Mini to a 430 GB EPYC server. + Full tables, methodology and quality ablations: docs/benchmarks.md

+
+ +
+
05 — the models
+

One engine, more minds coming

+

The tiering algorithm is model-agnostic: any Mixture-of-Experts with routed + experts can be staged the same way. GLM-5.2 is live today; support for more open-weight + families is on the roadmap.

+
+
live

GLM-5.2

Z.ai · 744B MoE
+

The flagship target: int4 container, token-exact vs reference, full atlas published.

+
live

OLMoE

Allen AI · 7B MoE
+

The small research workhorse — quantization A/Bs and quality ablations run here first.

+
planned

Kimi K2

Moonshot AI · 1T MoE
+

The next scale step: a trillion-parameter mind on the same tiered engine.

+
planned

Qwen3 MoE

Alibaba Qwen
+

The most widely-deployed open family — broad hardware coverage meets broad adoption.

+
planned

MiniMax

MiniMax · MoE
+

Long-context specialists — a different routing profile for the atlas to map.

+
+
+

Thank you to the teams whose open weights make this project possible — + Z.ai (GLM), Moonshot AI (Kimi), Alibaba Qwen, MiniMax, + Allen AI (OLMoE), and everyone releasing frontier-class models in the open. + An engine is nothing without a mind to run.

+
+
+ +
+
06 — join in
+

Come build the microscope

+

Every optimisation in this project started with someone measuring something on + their own machine — a disk swap, a NUMA experiment, an atlas replication on a Mac Mini. + You don't need a datacenter to move frontier-model research. That is the whole point.

+
+

▸ run a benchmark

+

Any hardware is a datapoint — the community table grows one issue at a time.

 +

▸ pick an issue

+

Kernels, schedulers, the atlas, the dashboard — measured problems, waiting.

 +

▸ read the engine

+

One C file plus headers, zero dependencies. Readable in an afternoon — really.

 +
+
+ +
+ + + + + +