url/colibri
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Dual-SSD streaming: COLI_MODEL_MIRROR reads experts from two model copies

Browse Source 
A second (read-only) copy of the model on another drive is registered as a
per-shard read replica; expert loads are split between the drives by a
deterministic (layer,eid) hash weighted by COLI_DISK_WEIGHTS=<primary>,<mirror>
or, by default, by a startup bandwidth probe using the engine's own access
pattern (parallel ~19 MB reads, O_DIRECT). Cold decode is disk-bound
(~11 GB/token), so two NVMe queues add up.

- st.h: mirror accepted per file only if size + safetensors header are
  byte-identical to the primary (identical data_offsets by construction, so
  every pread is valid on either copy); partial mirrors work (smaller second
  SSD holding only some shards); the mirror is never written — .coli_usage /
  .coli_kv stay on the primary.
- glm.c: routing covers the coalesced slab pread, O_DIRECT, mmap/Metal and
  scale reads, plus the OMP-parallel pin/autopin warmup (streams from both
  drives). Deterministic routing keeps readahead/PILOT WILLNEED on the same
  drive as the demand read and avoids caching an expert twice. A mirror read
  error falls back to the primary (one warning, no crash). Per-drive bytes
  are reported in a MIRROR: stats line.
- tests: test_st_mirror covers validation, read equality on both replicas,
  and the rejection paths (divergent header, size mismatch, missing file).

Measured on 2x NVMe (GLM-5.2 int4, greedy, DRAFT=0, DIRECT=1, cold-ish
cache): 0.42 -> 0.57 tok/s (+36%), expert-disk service 15.2s -> 10.3s,
byte-identical output; probe chose a 48/52 split.
This commit is contained in:
Steve Markgraf
2026-07-14 01:23:18 +02:00
parent 61004dcb84
commit 63966ba3f8
7 changed files with 370 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files
c/Makefile
+4 -1
View File
@@ -166,7 +166,7 @@ else
PYTHON ?= python3
endif
CUDA_OBJ =
TEST_BINS = tests/test_json$(EXE) tests/test_st$(EXE) tests/test_st_pread$(EXE) tests/test_tier$(EXE) tests/test_grammar$(EXE) tests/test_schema_gbnf$(EXE) tests/test_decode_batch$(EXE) tests/test_idot$(EXE) tests/test_i4_grouped$(EXE) tests/test_stops$(EXE) tests/test_topp$(EXE) tests/test_sample_nan$(EXE) tests/test_kv_alloc$(EXE) tests/test_i4_acc512$(EXE) tests/test_compat_direct$(EXE) tests/test_dsa_select$(EXE) tests/test_logit_nan$(EXE)
TEST_BINS = tests/test_json$(EXE) tests/test_st$(EXE) tests/test_st_mirror$(EXE) tests/test_st_pread$(EXE) tests/test_tier$(EXE) tests/test_grammar$(EXE) tests/test_schema_gbnf$(EXE) tests/test_decode_batch$(EXE) tests/test_idot$(EXE) tests/test_i4_grouped$(EXE) tests/test_stops$(EXE) tests/test_topp$(EXE) tests/test_sample_nan$(EXE) tests/test_kv_alloc$(EXE) tests/test_i4_acc512$(EXE) tests/test_compat_direct$(EXE) tests/test_dsa_select$(EXE) tests/test_logit_nan$(EXE)
ifneq (,$(LINUX))
TEST_BINS += tests/test_uring$(EXE)
endif
@@ -300,6 +300,9 @@ tests/test_st_pread$(EXE): tests/test_st_pread.c st.h json.h compat.h
tests/test_st$(EXE): tests/test_st.c st.h json.h compat.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
tests/test_st_mirror$(EXE): tests/test_st_mirror.c st.h json.h compat.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
tests/test_tier$(EXE): tests/test_tier.c tier.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)