README: restructure around the technical path — parameters and full results move to docs/, four new figures
This commit is contained in:
+100
@@ -0,0 +1,100 @@
|
||||
# OpenAI-compatible API, KV contexts & web UI
|
||||
|
||||
## `coli serve`
|
||||
|
||||
`coli serve` keeps one model process loaded and exposes a text-only
|
||||
OpenAI-compatible HTTP API. The gateway uses only the Python standard library;
|
||||
inference still runs in the same dependency-free C engine.
|
||||
|
||||
```bash
|
||||
cd c
|
||||
COLI_MODEL=/nvme/glm52_i4 COLI_API_KEY=local-secret ./coli serve \
|
||||
--host 127.0.0.1 --port 8000 --model-id glm-5.2-colibri
|
||||
|
||||
curl http://127.0.0.1:8000/v1/chat/completions \
|
||||
-H 'Authorization: Bearer local-secret' \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{
|
||||
"model": "glm-5.2-colibri",
|
||||
"messages": [{"role": "user", "content": "Hello"}],
|
||||
"stream": true
|
||||
}'
|
||||
```
|
||||
|
||||
Implemented endpoints are `GET /v1/models`, `GET /v1/models/{model}`,
|
||||
`POST /v1/chat/completions`, and legacy `POST /v1/completions`. Chat and
|
||||
completion requests support JSON responses, SSE streaming, usage counts,
|
||||
`max_tokens`/`max_completion_tokens`, `temperature`, and `top_p`. The extension
|
||||
`enable_thinking: true` enables GLM-5.2's reasoning block; the standard
|
||||
`reasoning_effort` field also enables it unless set to `none`.
|
||||
|
||||
The server is deliberately text-only and serves one generation at a time: the
|
||||
744B model stays in one persistent process, so concurrent HTTP requests queue
|
||||
instead of loading duplicate model copies. Tools, image/audio input, custom
|
||||
stop sequences, log probabilities, and token penalties return an explicit error
|
||||
rather than being silently ignored. The default bind address is localhost; set
|
||||
`COLI_API_KEY` before exposing the server beyond the machine.
|
||||
|
||||
Browser access from the Vite development server and Tauri local origins is
|
||||
enabled by default. Repeat `--cors-origin https://your-ui.example` to allow
|
||||
another exact origin, or use `--cors-origin '*'` only on a trusted local
|
||||
network.
|
||||
|
||||
The engine owns its KV contexts, so HTTP generation uses a bounded FIFO
|
||||
admission queue instead of pretending to run unsafe parallel sequences.
|
||||
Configure it with `--max-queue N` (default 8) and `--queue-timeout SECONDS`
|
||||
(default 300), or the `COLI_MAX_QUEUE` / `COLI_QUEUE_TIMEOUT` environment
|
||||
variables. Saturated and timed-out requests receive OpenAI-shaped HTTP 429
|
||||
errors before streaming headers are sent. `GET /health` exposes
|
||||
active/queued/completed/rejected counters, and successful generation responses
|
||||
include `x-colibri-queue-wait-ms`.
|
||||
|
||||
## Isolated KV contexts
|
||||
|
||||
`coli serve --kv-slots N` allocates up to 16 independent sequence contexts.
|
||||
Requests select one with the optional integer `cache_slot` field; ordinary
|
||||
OpenAI clients omit it and keep the original slot 0 behavior.
|
||||
|
||||
```json
|
||||
{
|
||||
"model": "glm-5.2-colibri",
|
||||
"messages": [{"role": "user", "content": "Continue this conversation"}],
|
||||
"cache_slot": 1
|
||||
}
|
||||
```
|
||||
|
||||
Each slot owns its token history, compressed MLA/DSA KV memory, MTP window, and
|
||||
crash-safe persistence file (`.coli_kv`, `.coli_kv.1`, ...). The engine matches
|
||||
each request's tokenized prompt against the slot's history and reuses the common
|
||||
KV prefix, so stateless HTTP turns keep their cache across requests and even
|
||||
across engine restarts. Use `COLI_KV_SLOTS=N` as the environment equivalent.
|
||||
Start small: at the default 4096-token context, every slot costs hundreds of MB.
|
||||
|
||||
## Web dashboard
|
||||
|
||||
One command serves the OpenAI-compatible API **and** the web console on the
|
||||
same port, then opens your browser when the engine is ready:
|
||||
|
||||
```bash
|
||||
cd web && npm install && npm run build # once
|
||||
./coli web --model <model-dir>
|
||||
```
|
||||
|
||||
What you get:
|
||||
|
||||
- **Chat** with live metrics: a flashing token counter while generating, then
|
||||
tok/s, time-to-first-token, prompt→completion counts and queue wait;
|
||||
- **Runtime panel**: your hardware (CPU, GPUs + VRAM, RAM, cores), the
|
||||
scheduler, and the live expert-tier bar — how many of the 19,456 experts sit
|
||||
in VRAM / RAM / disk right now;
|
||||
- **Brain**: the whole model as a 76×256 cortex, one cell per expert. Colour =
|
||||
tier, brightness = routing heat, and the experts routed in each turn flash
|
||||
white and decay — you watch the model think. Hover any cell for its tier,
|
||||
heat and [measured topic affinity](https://github.com/JustVugg/colibri/issues/175);
|
||||
- **Atlas**: the measured expert atlas as a 3-D galaxy (publish `experts.json`
|
||||
from `tools/expert_atlas/analyze.py --web`).
|
||||
|
||||
The dashboard talks to the engine over a small line protocol and plain JSON
|
||||
endpoints — nothing heavier than the engine itself. `web/` is a pure OpenAI-API
|
||||
client (React + TypeScript) and also works against any other compatible
|
||||
endpoint; the terminal `coli chat` remains the first-class interface.
|
||||
Reference in New Issue
Block a user