feat(st): COLI_MODEL_DIRS - split model shards across N drives, no duplication
Each extra directory holds a DISTINCT subset of the .safetensors shards (search path, dedup by basename; first-listed dir wins). Demand preads hit whichever drive holds the shard, so concurrent expert loads parallelise across drives and combined capacity is used - a 400 GB container fits across two smaller drives that individually cannot hold it, which COLI_MODEL_MIRROR (a full second copy) cannot do. Composable with the mirror: st_mirror_init matches per-shard by basename against the merged index. st_init stays as a back-compat wrapper over st_init_multi. Verified on RTX 5080 / Windows: 72+70 shards across two NVMes, coherent output, [SPLIT] startup log, decode parity with single-drive at full RAM (0.92 vs 0.89-0.90).
This commit is contained in:
+3
-1
@@ -1031,7 +1031,9 @@ static void layer_cuda_shard_kvb(Layer *l,int H,int Q,int V){
|
||||
|
||||
static void model_init(Model *m, const char *snap, int cap, int ebits, int dbits){
|
||||
memset(m,0,sizeof(*m)); m->ebits=ebits; m->dbits=dbits;
|
||||
load_cfg(&m->c,snap); st_init(&m->S,snap);
|
||||
load_cfg(&m->c,snap);
|
||||
{ const char *xd=getenv("COLI_MODEL_DIRS"); /* SPLIT: model shards spread across N drives */
|
||||
st_init_multi(&m->S,snap,(xd&&*xd)?xd:NULL); }
|
||||
Cfg *c=&m->c; char nm[256]; int H=c->n_heads, D=c->hidden;
|
||||
/* embed e lm_head sono il confine I/O: tenerli ad alta precisione (come i quant dynamic
|
||||
* reali). A bf16 ~1.9GB su GLM reale: trascurabile. dbits>=8 -> qui f32; piu' basso -> dbits. */
|
||||
|
||||
Reference in New Issue
Block a user