url/colibri
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge #469: COLI_MODEL_DIRS — split model shards across N drives (capacity aggregation)

Browse Source 
Search-path of extra directories, each holding a distinct subset of the .safetensors shards (no duplication). Each shard lives on one drive; demand preads hit whichever drive holds it, so concurrent expert loads parallelise across drives and combined capacity is used — a 400 GB container fits across two 250 GB drives the mirror (#421) can't. st_init kept as a back-compat wrapper. Verified locally: clean build, no regression on the default path (token-exact tiny models unchanged), and the split path itself validated token-exact (shard on a separate dir, metadata in primary -> 32/32 vs oracle). Composable with the #421 mirror. CI 8/8. Thanks @mohamedmastouri2000-boop.
This commit is contained in:
Vincenzo Fornaro
2026-07-21 00:57:16 +02:00
committed by GitHub
parent 4cc9885cb3 0d2fb6f8a2
commit e16ec68168
3 changed files with 61 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/ENVIRONMENT.md
+1
View File
@@ -82,6 +82,7 @@ Format: `VAR` — default — effect.
| Variable | Default | Effect |
|---|---|---|
| `COLI_MODEL_DIRS` | unset | SPLIT the model across 2+ drives: a `;`/`,`-separated list of extra directories, each holding a **distinct** subset of the `.safetensors` shards (no duplication). Shards act as a search path — every shard is read from whichever drive holds it, so concurrent expert loads parallelise across drives and combined capacity is used. Scales to N drives. Metadata (config/tokenizer/`.coli_usage`) stays in the primary `COLI_MODEL` dir. Pairs well with `PIPE=1` (concurrent loaders) + `DIRECT=1`. Distinct from — and composable with — `COLI_MODEL_MIRROR`: the mirror is matched per-shard by basename against the merged (split) index, so a mirror dir may hold a copy of any subset of the split's shards. |
| `COLI_MODEL_MIRROR` | unset | Path to a second, byte-identical (read-only) copy of the model on another drive; expert reads are split across both. Partial mirrors work (only the shards present are used). |
| `COLI_DISK_WEIGHTS` | unset (startup bandwidth probe) | Split ratio `<primary>,<mirror>` (e.g. `1,1` for 50/50, `9,3` for a fast+slow pair). Unset = probe both drives with the engine's own access pattern at startup. |