ZacharyZcR
111 lines
4.8 KiB
Markdown
111 lines
4.8 KiB
Markdown
# OpenAI-compatible API, KV contexts & web UI
|
||
|
||
## `coli serve`
|
||
|
||
`coli serve` keeps one model process loaded and exposes a text-only
|
||
OpenAI-compatible HTTP API. The gateway uses only the Python standard library;
|
||
inference still runs in the same dependency-free C engine.
|
||
|
||
```bash
|
||
cd c
|
||
COLI_MODEL=/nvme/glm52_i4 COLI_API_KEY=local-secret ./coli serve \
|
||
--host 127.0.0.1 --port 8000 --model-id glm-5.2-colibri
|
||
|
||
curl http://127.0.0.1:8000/v1/chat/completions \
|
||
-H 'Authorization: Bearer local-secret' \
|
||
-H 'Content-Type: application/json' \
|
||
-d '{
|
||
"model": "glm-5.2-colibri",
|
||
"messages": [{"role": "user", "content": "Hello"}],
|
||
"stream": true
|
||
}'
|
||
```
|
||
|
||
Implemented endpoints are `GET /v1/models`, `GET /v1/models/{model}`,
|
||
`POST /v1/chat/completions`, and legacy `POST /v1/completions`. Chat and
|
||
completion requests support JSON responses, SSE streaming, usage counts,
|
||
`max_tokens`/`max_completion_tokens`, `temperature`, and `top_p`. The extension
|
||
`enable_thinking: true` enables GLM-5.2's reasoning block; the standard
|
||
`reasoning_effort` field also enables it unless set to `none`.
|
||
|
||
The server is deliberately text-only and serves one generation at a time: the
|
||
744B model stays in one persistent process, so concurrent HTTP requests queue
|
||
instead of loading duplicate model copies. Tools, image/audio input, custom
|
||
stop sequences, log probabilities, and token penalties return an explicit error
|
||
rather than being silently ignored. The default bind address is localhost; set
|
||
`COLI_API_KEY` before exposing the server beyond the machine.
|
||
|
||
Browser access from the Vite development server and Tauri local origins is
|
||
enabled by default. Repeat `--cors-origin https://your-ui.example` to allow
|
||
another exact origin, or use `--cors-origin '*'` only on a trusted local
|
||
network.
|
||
|
||
The engine owns its KV contexts, so HTTP generation uses a bounded FIFO
|
||
admission queue instead of pretending to run unsafe parallel sequences.
|
||
Configure it with `--max-queue N` (default 8) and `--queue-timeout SECONDS`
|
||
(default 300), or the `COLI_MAX_QUEUE` / `COLI_QUEUE_TIMEOUT` environment
|
||
variables. Saturated and timed-out requests receive OpenAI-shaped HTTP 429
|
||
errors before streaming headers are sent. `GET /health` exposes
|
||
active/queued/completed/rejected counters, and successful generation responses
|
||
include `x-colibri-queue-wait-ms`.
|
||
|
||
## Isolated KV contexts
|
||
|
||
`coli serve --kv-slots N` allocates up to 16 independent sequence contexts.
|
||
Requests select one with the optional integer `cache_slot` field; ordinary
|
||
OpenAI clients omit it and keep the original slot 0 behavior.
|
||
|
||
```json
|
||
{
|
||
"model": "glm-5.2-colibri",
|
||
"messages": [{"role": "user", "content": "Continue this conversation"}],
|
||
"cache_slot": 1
|
||
}
|
||
```
|
||
|
||
Each slot owns its token history, compressed MLA/DSA KV memory, MTP window, and
|
||
crash-safe persistence file (`.coli_kv`, `.coli_kv.1`, ...). The engine matches
|
||
each request's tokenized prompt against the slot's history and reuses the common
|
||
KV prefix, so stateless HTTP turns keep their cache across requests and even
|
||
across engine restarts. Use `COLI_KV_SLOTS=N` as the environment equivalent.
|
||
Start small: at the default 4096-token context, every slot costs hundreds of MB.
|
||
|
||
## Web dashboard
|
||
|
||
One command serves the OpenAI-compatible API **and** the web console on the
|
||
same port, then opens your browser when the engine is ready:
|
||
|
||
```bash
|
||
cd web && npm install && npm run build # once
|
||
./coli web --model <model-dir>
|
||
```
|
||
|
||
What you get:
|
||
|
||
- **Chat** with live metrics: a flashing token counter while generating, then
|
||
tok/s, time-to-first-token, prompt→completion counts and queue wait;
|
||
- **Runtime panel**: your hardware (CPU, GPUs + VRAM, RAM, cores), the
|
||
scheduler, and the live expert-tier bar — how many of the 19,456 experts sit
|
||
in VRAM / RAM / disk right now;
|
||
- **Brain**: the whole model as a 76×256 cortex, one cell per expert. Colour =
|
||
tier, brightness = routing heat, and the experts routed in each turn flash
|
||
white and decay — you watch the model think. Hover any cell for its tier,
|
||
heat and [measured topic affinity](https://github.com/JustVugg/colibri/issues/175);
|
||
- **Atlas**: the measured expert atlas as a 3-D galaxy (publish `experts.json`
|
||
from `tools/expert_atlas/analyze.py --web`).
|
||
|
||
The dashboard talks to the engine over a small line protocol and plain JSON
|
||
endpoints — nothing heavier than the engine itself. `web/` is a pure OpenAI-API
|
||
client (React + TypeScript) and also works against any other compatible
|
||
endpoint; the terminal `coli chat` remains the first-class interface.
|
||
|
||
The layout is responsive down to phone widths, and the sidebar carries the full
|
||
telemetry stack — hardware, scheduler, tier bar, per-turn time breakdown, tok/s
|
||
trend and per-GPU expert counts:
|
||
|
||
<p align="center">
|
||
<img src="media/colibri-mobile.png" width="270" alt="the dashboard on a phone-sized viewport">
|
||
|
||
<img src="media/colibri-metrics.png" width="300" alt="the telemetry sidebar">
|
||
</p>
|