feat(st): COLI_MODEL_DIRS - split model shards across N drives, no duplication

Each extra directory holds a DISTINCT subset of the .safetensors shards (search path, dedup by basename; first-listed dir wins). Demand preads hit whichever drive holds the shard, so concurrent expert loads parallelise across drives and combined capacity is used - a 400 GB container fits across two smaller drives that individually cannot hold it, which COLI_MODEL_MIRROR (a full second copy) cannot do. Composable with the mirror: st_mirror_init matches per-shard by basename against the merged index. st_init stays as a back-compat wrapper over st_init_multi. Verified on RTX 5080 / Windows: 72+70 shards across two NVMes, coherent output, [SPLIT] startup log, decode parity with single-drive at full RAM (0.92 vs 0.89-0.90).
This commit is contained in:
Mohamed Mastouri
2026-07-21 01:42:04 +03:00
parent 68ac9ff696
commit 0d2fb6f8a2
3 changed files with 61 additions and 10 deletions
+3 -1
View File
@@ -1031,7 +1031,9 @@ static void layer_cuda_shard_kvb(Layer *l,int H,int Q,int V){
static void model_init(Model *m, const char *snap, int cap, int ebits, int dbits){
memset(m,0,sizeof(*m)); m->ebits=ebits; m->dbits=dbits;
load_cfg(&m->c,snap); st_init(&m->S,snap);
load_cfg(&m->c,snap);
{ const char *xd=getenv("COLI_MODEL_DIRS"); /* SPLIT: model shards spread across N drives */
st_init_multi(&m->S,snap,(xd&&*xd)?xd:NULL); }
Cfg *c=&m->c; char nm[256]; int H=c->n_heads, D=c->hidden;
/* embed e lm_head sono il confine I/O: tenerli ad alta precisione (come i quant dynamic
* reali). A bf16 ~1.9GB su GLM reale: trascurabile. dbits>=8 -> qui f32; piu' basso -> dbits. */