colibri/docs/tuning.md

Tuning & runtime knobs

Everything here is opt-in; the defaults are chosen so a plain ./coli chat is safe on any machine. See also SETTINGS.md and ENVIRONMENT.md for the full variable inventory.

The knobs that matter most

knob	what it does
--temp T	token sampling temperature (default 0.7 + nucleus 0.90 — tuned for int4; 0 = greedy)
--topp 0.7	adaptive expert top-p (30–40% less disk; lossy — prints a warning)
--ngen N	max tokens per answer (:more in chat continues a truncated one)
--repin N	adapt RAM/VRAM hot experts every N emitted tokens
RAM_GB=<n>	claim more RAM for the expert cache than the conservative auto-detect
PIN=stats PIN_GB=g	pin the hottest experts from a measured usage profile
DRAFT=n	MTP draft depth (0 disables speculation)
GRAMMAR=g.gbnf	grammar-forced drafts for constrained JSON/NDJSON output (docs)
THINK=1	enable GLM-5.2's reasoning block
PILOT=1	router-lookahead disk prefetch (see below)
URING=1	Linux-only batched expert I/O (implies PIPE=1)
PIPE=0	disable the async expert-load pool (default ON — overlaps pread with matmul, −18% disk service)
DIRECT=1	O_DIRECT expert reads (measured +65% alone on a Strix Halo, #200)
COLI_NUMA=1	interleave resident weights across NUMA nodes on multi-socket hosts (#82)
CACHE_ROUTE=1	cache-aware max-rank routing (opt-in, #199)
AUTOPIN=0	disable the learning cache's auto-pin
CAP_RAISE=0	don't auto-grow the expert cache
KVSAVE=0	disable KV-cache persistence
TF=1	teacher-forcing validation

Resource policy

coli plan reports the planned hot (VRAM), warm (RAM), and cold backing (disk) tiers, the reason for each placement, and the expected bottleneck. The default --policy quality and --policy balanced modes preserve checkpoint quantization and router decisions unless --topk or --topp is passed; those explicit lossy overrides print a warning and proceed.

Auto-tier plans size OpenMP from physical cores and bind workers across cores. Memory-bound quantized kernels can regress sharply when SMT siblings compete for limited memory channels; explicit OMP_* settings always take precedence.

coli plan --model /models/glm52_i4 --policy quality
coli run --auto-tier --policy quality "Explain MoE offloading"
# Explicit research-only router reduction:
coli run --policy experimental-fast --topk 4 "Benchmark prompt"

Disk is an immutable recovery source, not a normal decode target. If the plan leaves cold expert bytes on disk, speed depends on cache hit rate; output quality does not.

Cold expert reads can use a deferred pipeline: resident RAM/VRAM experts execute while missing experts are loaded in a bounded background I/O pool, then the cold results join before the layer completes. The pool engages only under PIPE=1; PIPE_WORKERS=n sets its worker count (default 8). Profiling reports both disk service time and the smaller foreground-visible wait time so overlap is explicit.

--policy balanced enables lossless live placement (REPIN=64). At safe request boundaries, a per-layer LFRU score combines decaying session frequency with recent access and replaces at most four sufficiently colder pinned experts. --policy quality leaves live replacement off by default; REPIN=0 always disables it.

The learning cache

The engine records which experts your usage actually routes to (.coli_usage next to the model, updated every turn) and at startup automatically pins the hottest ones in spare RAM — colibrì literally gets faster the more you use it. PIN=auto seeds the pin directly from the live usage history (#301).

The expert cache auto-sizes to your RAM (since 2026-07-10): the engine raises the LRU cap to fill your --ram budget instead of only lowering it. If you benchmarked colibrì before that date, rerun — your numbers were capped.

Live tier adaptation (--repin N, opt-in): at safe turn boundaries, a decaying session heat map replaces cold pinned experts with hotter streamed experts. A 25% hysteresis and a four-swap limit prevent tier thrashing. Persistent .coli_usage remains the long-term signal and is not decayed.

Router-lookahead prefetch (PILOT=1, experimental)

GLM-5.2's expert routing is measurably predictable ahead of time — applying layer L+1's router to layer L's post-attention state recalls 71.6% of the true top-8 (vs 41.3% for "same experts as last token"). PILOT=1 issues next-layer expert readahead from a dedicated I/O thread while the current layer computes. PILOT_REAL=1 moves the prefetched loads off the critical path (measured +11pp hit rate on a big-cache host), and PILOT_TWO=1 folds the computed shared-expert into the prediction (+3% recall, #200). On disk-saturated hosts hint-only PILOT can be net negative — measure on yours.

Speculation and reproducibility

Speculative decoding requires that the draft and verify paths compute the same function — SPEC_PIN=1 (default since #294) pins every forward issued while drafts are live to the platform's S=1 kernel family. For byte-exact reproducibility across runs: DRAFT=0, plus IDOT=0 COLI_CUDA=0 if you also want kernel-family/GPU independence. Acceptance percentages are not comparable across engine versions under --topp (#163 has the full story).

Conversations reopen warm

coli chat persists the compressed MLA KV-cache to disk after every turn (.coli_kv, ~182 KB/token, appended incrementally, crash-safe). Close the chat, reopen it tomorrow — the model still remembers the whole conversation and zero re-prefill happens: validated byte-identical to an uninterrupted session. :reset clears it, KVSAVE=0 disables it.