Merge dev into dual-ssd-mirror: resolve #298/#192-era conflicts
Combined resolution: mir_pread/st_prefetch_rep (this PR) now carry dev's DISK-CLASS accounting unwind (dc_wall_exit) and O_DIRECT prefetch skip (g_direct); direct-path keeps dc_direct=1 plus the mirror read counters. Verified: clean build, token-exact tiny models unchanged, test_st_mirror and test_st_pread pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -114,6 +114,15 @@ Per-drive byte counts are reported in a `MIRROR:` stats line. Combine with `DIRE
|
||||
| `COLI_CUDA_SHARED_W4A16_MIN_ROWS` | `32` | Min row count to engage the shared-MLP W4A16 kernel. |
|
||||
| `COLI_METAL_UNTRACKED` | off (Metal only) | `=1` sets `MTLResourceHazardTrackingModeUntracked` on Metal buffers (reduces hazard-tracking overhead). |
|
||||
|
||||
> **Windows note.** On Windows, a bare `coli chat` / `coli run` / `coli serve`
|
||||
> (no `--gpu`/`--vram`/`--auto-tier`) **auto-enables the GPU** when it detects a
|
||||
> CUDA build (`coli_cuda.dll` next to the engine) and at least one GPU via
|
||||
> `nvidia-smi`. The expert-tier VRAM budget is then sized automatically from the
|
||||
> card's free VRAM (same computation as `--auto-tier`). If `nvidia-smi` is not on
|
||||
> `PATH` the run falls back to CPU with a warning — pass `--vram N` (or add
|
||||
> `nvidia-smi` to `PATH`) to enable CUDA in that case. `--gpu none` forces
|
||||
> CPU-only. (Linux/macOS behaviour is unchanged: pass a flag to enable CUDA.)
|
||||
|
||||
---
|
||||
|
||||
## Advanced / experimental / debug
|
||||
|
||||
@@ -83,3 +83,27 @@ this implementation adds: forced spans as a **draft source verified in the targe
|
||||
own forward** (lossless even under a wrong grammar, composes with MTP/n-gram in one
|
||||
union batch), deployed where the win is denominated in expert I/O rather than
|
||||
forward passes.
|
||||
|
||||
## Server usage: `response_format` (OpenAI API)
|
||||
|
||||
The gateway (`openai_server.py`) turns `response_format` into a **per-request**
|
||||
grammar carried to the engine over the `SUBMIT` protocol (optional 7th header
|
||||
field, `gbytes`; 6-field headers from older clients remain valid — the field is
|
||||
additive and back-compatible in both directions):
|
||||
|
||||
```jsonc
|
||||
{"response_format": {"type": "json_object"}} // generic JSON grammar
|
||||
{"response_format": {"type": "json_schema",
|
||||
"json_schema": {"schema": { ... }}}} // compiled by schema_gbnf.h
|
||||
{"response_format": {"type": "gbnf", "grammar": "root ::= ..."}} // raw GBNF (extension)
|
||||
```
|
||||
|
||||
Semantics are identical to `GRAMMAR=`/`SCHEMA=`: a **draft source, never a
|
||||
sampling constraint**. A schema outside the supported subset, or malformed GBNF,
|
||||
costs the speedup — never the request, never the output. Drafting engages for
|
||||
greedy requests (`temperature: 0`); sampled requests run undrafted. Compile
|
||||
overhead is negligible: ~8 µs/request for a typical schema, ~18 µs at the
|
||||
32-level nesting cap (measured, M3 Max). Grammar payloads are capped at 1 MiB.
|
||||
As with MTP (#100), a drafted greedy run may differ from an undrafted one in
|
||||
near-tie tokens (the verify forward has a different batch shape); each output
|
||||
is a valid greedy stream of its own forward shapes.
|
||||
|
||||
+25
-5
@@ -42,11 +42,31 @@ engine needs.
|
||||
You have two options.
|
||||
|
||||
**Option A — download a prebuilt binary (no compiler needed).**
|
||||
Grab the latest `colibri-<version>-windows-x86_64.zip` from the
|
||||
[Releases page](https://github.com/JustVugg/colibri/releases), unzip it, and
|
||||
skip to [step 3](#3-get-the-model). Python 3 (from
|
||||
[python.org](https://www.python.org/downloads/)) is still needed if you want to
|
||||
convert a model yourself.
|
||||
Grab `colibri-<version>-windows-x86_64.zip` from the
|
||||
[Releases page](https://github.com/JustVugg/colibri/releases) and unzip it.
|
||||
Inside you'll find:
|
||||
|
||||
| File | What it is |
|
||||
|---|---|
|
||||
| `colibri-<version>-windows-x86_64.exe` | **the engine** — the C program that actually runs the model |
|
||||
| `coli` | the command-line launcher (`chat`, `serve`, `convert`, `doctor`, …) |
|
||||
| `openai_server.py`, `resource_plan.py`, `doctor.py` | Python support for the API server and placement planner |
|
||||
|
||||
Two setup steps:
|
||||
|
||||
1. **Rename the engine to `glm.exe`** so the launcher can find it (it looks for a
|
||||
binary named `glm`):
|
||||
```powershell
|
||||
Rename-Item colibri-*-windows-x86_64.exe glm.exe
|
||||
```
|
||||
2. **Install Python 3** from [python.org](https://www.python.org/downloads/) — the
|
||||
`coli` launcher and the API gateway are Python scripts (the engine itself is
|
||||
pure C and needs nothing).
|
||||
|
||||
Then continue to [step 3](#3-get-the-model). Prefer to skip the launcher? You can
|
||||
run the engine directly — `.\glm.exe` reads the model path from the `SNAP`
|
||||
environment variable (see [docs/windows.md](windows.md)) — but `coli chat` is the
|
||||
easy path.
|
||||
|
||||
**Option B — build from source with MSYS2.**
|
||||
Install [MSYS2](https://www.msys2.org/), open the **UCRT64** shell, and run:
|
||||
|
||||
Reference in New Issue
Block a user