Merge dev into dual-ssd-mirror: resolve #298/#192-era conflicts

Combined resolution: mir_pread/st_prefetch_rep (this PR) now carry dev's
DISK-CLASS accounting unwind (dc_wall_exit) and O_DIRECT prefetch skip
(g_direct); direct-path keeps dc_direct=1 plus the mirror read counters.
Verified: clean build, token-exact tiny models unchanged, test_st_mirror
and test_st_pread pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
JustVugg
2026-07-20 21:24:39 +02:00
60 changed files with 8507 additions and 346 deletions
+9
View File
@@ -114,6 +114,15 @@ Per-drive byte counts are reported in a `MIRROR:` stats line. Combine with `DIRE
| `COLI_CUDA_SHARED_W4A16_MIN_ROWS` | `32` | Min row count to engage the shared-MLP W4A16 kernel. |
| `COLI_METAL_UNTRACKED` | off (Metal only) | `=1` sets `MTLResourceHazardTrackingModeUntracked` on Metal buffers (reduces hazard-tracking overhead). |
> **Windows note.** On Windows, a bare `coli chat` / `coli run` / `coli serve`
> (no `--gpu`/`--vram`/`--auto-tier`) **auto-enables the GPU** when it detects a
> CUDA build (`coli_cuda.dll` next to the engine) and at least one GPU via
> `nvidia-smi`. The expert-tier VRAM budget is then sized automatically from the
> card's free VRAM (same computation as `--auto-tier`). If `nvidia-smi` is not on
> `PATH` the run falls back to CPU with a warning — pass `--vram N` (or add
> `nvidia-smi` to `PATH`) to enable CUDA in that case. `--gpu none` forces
> CPU-only. (Linux/macOS behaviour is unchanged: pass a flag to enable CUDA.)
---
## Advanced / experimental / debug
+24
View File
@@ -83,3 +83,27 @@ this implementation adds: forced spans as a **draft source verified in the targe
own forward** (lossless even under a wrong grammar, composes with MTP/n-gram in one
union batch), deployed where the win is denominated in expert I/O rather than
forward passes.
## Server usage: `response_format` (OpenAI API)
The gateway (`openai_server.py`) turns `response_format` into a **per-request**
grammar carried to the engine over the `SUBMIT` protocol (optional 7th header
field, `gbytes`; 6-field headers from older clients remain valid — the field is
additive and back-compatible in both directions):
```jsonc
{"response_format": {"type": "json_object"}} // generic JSON grammar
{"response_format": {"type": "json_schema",
"json_schema": {"schema": { ... }}}} // compiled by schema_gbnf.h
{"response_format": {"type": "gbnf", "grammar": "root ::= ..."}} // raw GBNF (extension)
```
Semantics are identical to `GRAMMAR=`/`SCHEMA=`: a **draft source, never a
sampling constraint**. A schema outside the supported subset, or malformed GBNF,
costs the speedup — never the request, never the output. Drafting engages for
greedy requests (`temperature: 0`); sampled requests run undrafted. Compile
overhead is negligible: ~8 µs/request for a typical schema, ~18 µs at the
32-level nesting cap (measured, M3 Max). Grammar payloads are capped at 1 MiB.
As with MTP (#100), a drafted greedy run may differ from an undrafted one in
near-tie tokens (the verify forward has a different batch shape); each output
is a valid greedy stream of its own forward shapes.
+25 -5
View File
@@ -42,11 +42,31 @@ engine needs.
You have two options.
**Option A — download a prebuilt binary (no compiler needed).**
Grab the latest `colibri-<version>-windows-x86_64.zip` from the
[Releases page](https://github.com/JustVugg/colibri/releases), unzip it, and
skip to [step 3](#3-get-the-model). Python 3 (from
[python.org](https://www.python.org/downloads/)) is still needed if you want to
convert a model yourself.
Grab `colibri-<version>-windows-x86_64.zip` from the
[Releases page](https://github.com/JustVugg/colibri/releases) and unzip it.
Inside you'll find:
| File | What it is |
|---|---|
| `colibri-<version>-windows-x86_64.exe` | **the engine** — the C program that actually runs the model |
| `coli` | the command-line launcher (`chat`, `serve`, `convert`, `doctor`, …) |
| `openai_server.py`, `resource_plan.py`, `doctor.py` | Python support for the API server and placement planner |
Two setup steps:
1. **Rename the engine to `glm.exe`** so the launcher can find it (it looks for a
binary named `glm`):
```powershell
Rename-Item colibri-*-windows-x86_64.exe glm.exe
```
2. **Install Python 3** from [python.org](https://www.python.org/downloads/) — the
`coli` launcher and the API gateway are Python scripts (the engine itself is
pure C and needs nothing).
Then continue to [step 3](#3-get-the-model). Prefer to skip the launcher? You can
run the engine directly — `.\glm.exe` reads the model path from the `SNAP`
environment variable (see [docs/windows.md](windows.md)) — but `coli chat` is the
easy path.
**Option B — build from source with MSYS2.**
Install [MSYS2](https://www.msys2.org/), open the **UCRT64** shell, and run: