+
A compiler JIT never compiles the whole program — it watches what actually runs
+ and compiles the hot paths, just in time. colibrì applies the same bet to a
+ 744B parameter space: a token activates only ~40B parameters, and just
+ ~11 GB of those change from token to token — the routed experts.
+
So parameters are not resident state to be held — they are
+ data to be staged, exactly when the router proves they are needed.
+
The engine keeps the dense weights resident and treats the 19,456 experts as a
+ heterogeneous storage hierarchy: measured routing heat decides which experts earn
+ VRAM, which earn pinned RAM, and which stream from NVMe. The router runs a layer
+ ahead, so prefetch hides the staging latency — and because routing has
+ measurable structure (the atlas below), the hierarchy keeps getting the answer right.
+ Like a JIT, it learns your workload: the more you run, the hotter the right experts get.
+
Everything is validated token-exact against the reference transformers
+ implementation — speed never buys drift.
+
⚗ Active research: the next algorithmic step is under way —
+ smarter placement and scheduling beyond LRU + pin, overlap of CPU and GPU expert execution,
+ and routing-aware speculation. Measured, reviewed, and merged in the open.
+
+
+ VRAMhottest experts · grouped GPU matmuls ·
+ six 5090s hold the entire routed set — disk reads hit zero
+
↑ promote by measured heat ↓ demote
+
+ RAMwarm set · pinned & wired, learned from
+ your usage · AVX-512/NEON int4 kernels + LRU behind it
+
↑ promote ↓ demote
+
+ NVMecold tail · streamed on demand with io_uring,
+ prefetched a layer ahead — how 744B fits a 25 GB machine at all
+
token → router picks 8 of 256 experts × 75 layers →
+ gather across tiers → int4 matmul → next token
+