Dual-SSD streaming: COLI_MODEL_MIRROR reads experts from two model copies
A second (read-only) copy of the model on another drive is registered as a per-shard read replica; expert loads are split between the drives by a deterministic (layer,eid) hash weighted by COLI_DISK_WEIGHTS=<primary>,<mirror> or, by default, by a startup bandwidth probe using the engine's own access pattern (parallel ~19 MB reads, O_DIRECT). Cold decode is disk-bound (~11 GB/token), so two NVMe queues add up. - st.h: mirror accepted per file only if size + safetensors header are byte-identical to the primary (identical data_offsets by construction, so every pread is valid on either copy); partial mirrors work (smaller second SSD holding only some shards); the mirror is never written — .coli_usage / .coli_kv stay on the primary. - glm.c: routing covers the coalesced slab pread, O_DIRECT, mmap/Metal and scale reads, plus the OMP-parallel pin/autopin warmup (streams from both drives). Deterministic routing keeps readahead/PILOT WILLNEED on the same drive as the demand read and avoids caching an expert twice. A mirror read error falls back to the primary (one warning, no crash). Per-drive bytes are reported in a MIRROR: stats line. - tests: test_st_mirror covers validation, read equality on both replicas, and the rejection paths (divergent header, size mismatch, missing file). Measured on 2x NVMe (GLM-5.2 int4, greedy, DRAFT=0, DIRECT=1, cold-ish cache): 0.42 -> 0.57 tok/s (+36%), expert-disk service 15.2s -> 10.3s, byte-identical output; probe chose a 48/52 split.
This commit is contained in:
@@ -15,6 +15,9 @@ Run GLM-5.2 (744B) locally on CPU with roughly 15-26 GB of RAM.
|
||||
|
||||
Configuration through environment variables or flags (also valid after the subcommand):
|
||||
COLI_MODEL=<dir> model directory (default /home/vincenzo/glm52_i4)
|
||||
COLI_MODEL_MIRROR=<dir> second copy of the model on another drive: expert reads
|
||||
are split across both SSDs (COLI_DISK_WEIGHTS=9,3 sets the
|
||||
primary,mirror bandwidth ratio; default: measured at startup)
|
||||
--ram N RAM budget in GB (automatically sizes the expert cache)
|
||||
--repin N adapt RAM/VRAM experts every N tokens
|
||||
--topp P adaptive expert top-p --topk N fixed top-k
|
||||
|
||||
Reference in New Issue
Block a user