url/colibri
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(st): COLI_MODEL_DIRS - split model shards across N drives, no duplication

Browse Source 
Each extra directory holds a DISTINCT subset of the .safetensors shards (search path, dedup by basename; first-listed dir wins). Demand preads hit whichever drive holds the shard, so concurrent expert loads parallelise across drives and combined capacity is used - a 400 GB container fits across two smaller drives that individually cannot hold it, which COLI_MODEL_MIRROR (a full second copy) cannot do. Composable with the mirror: st_mirror_init matches per-shard by basename against the merged index. st_init stays as a back-compat wrapper over st_init_multi. Verified on RTX 5080 / Windows: 72+70 shards across two NVMes, coherent output, [SPLIT] startup log, decode parity with single-drive at full RAM (0.92 vs 0.89-0.90).
This commit is contained in:
Mohamed Mastouri
2026-07-21 01:42:04 +03:00
parent 68ac9ff696
commit 0d2fb6f8a2
3 changed files with 61 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/ENVIRONMENT.md
+1
View File
@@ -82,6 +82,7 @@ Format: `VAR` — default — effect.
| Variable | Default | Effect |
|---|---|---|
| `COLI_MODEL_DIRS` | unset | SPLIT the model across 2+ drives: a `;`/`,`-separated list of extra directories, each holding a **distinct** subset of the `.safetensors` shards (no duplication). Shards act as a search path — every shard is read from whichever drive holds it, so concurrent expert loads parallelise across drives and combined capacity is used. Scales to N drives. Metadata (config/tokenizer/`.coli_usage`) stays in the primary `COLI_MODEL` dir. Pairs well with `PIPE=1` (concurrent loaders) + `DIRECT=1`. Distinct from — and composable with — `COLI_MODEL_MIRROR`: the mirror is matched per-shard by basename against the merged (split) index, so a mirror dir may hold a copy of any subset of the split's shards. |
| `COLI_MODEL_MIRROR` | unset | Path to a second, byte-identical (read-only) copy of the model on another drive; expert reads are split across both. Partial mirrors work (only the shards present are used). |
| `COLI_DISK_WEIGHTS` | unset (startup bandwidth probe) | Split ratio `<primary>,<mirror>` (e.g. `1,1` for 50/50, `9,3` for a fast+slow pair). Unset = probe both drives with the engine's own access pattern at startup. |