colibri/docs/api.md

OpenAI-compatible API, KV contexts & web UI

coli serve

coli serve keeps one model process loaded and exposes a text-only OpenAI-compatible HTTP API. The gateway uses only the Python standard library; inference still runs in the same dependency-free C engine.

cd c
COLI_MODEL=/nvme/glm52_i4 COLI_API_KEY=local-secret ./coli serve \
  --host 127.0.0.1 --port 8000 --model-id glm-5.2-colibri

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Authorization: Bearer local-secret' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "glm-5.2-colibri",
    "messages": [{"role": "user", "content": "Hello"}],
    "stream": true
  }'

Implemented endpoints are GET /v1/models, GET /v1/models/{model}, POST /v1/chat/completions, and legacy POST /v1/completions. Chat and completion requests support JSON responses, SSE streaming, usage counts, max_tokens/max_completion_tokens, temperature, and top_p. The extension enable_thinking: true enables GLM-5.2's reasoning block; the standard reasoning_effort field also enables it unless set to none.

The server is deliberately text-only and serves one generation at a time: the 744B model stays in one persistent process, so concurrent HTTP requests queue instead of loading duplicate model copies. Tools, image/audio input, custom stop sequences, log probabilities, and token penalties return an explicit error rather than being silently ignored. The default bind address is localhost; set COLI_API_KEY before exposing the server beyond the machine.

Browser access from the Vite development server and Tauri local origins is enabled by default. Repeat --cors-origin https://your-ui.example to allow another exact origin, or use --cors-origin '*' only on a trusted local network.

The engine owns its KV contexts, so HTTP generation uses a bounded FIFO admission queue instead of pretending to run unsafe parallel sequences. Configure it with --max-queue N (default 8) and --queue-timeout SECONDS (default 300), or the COLI_MAX_QUEUE / COLI_QUEUE_TIMEOUT environment variables. Saturated and timed-out requests receive OpenAI-shaped HTTP 429 errors before streaming headers are sent. GET /health exposes active/queued/completed/rejected counters, and successful generation responses include x-colibri-queue-wait-ms.

Connect a coding CLI or editor

The API is OpenAI-compatible, so most coding CLIs and editor extensions work by pointing them at Colibri as an OpenAI-compatible provider. Three settings:

• Base URL — http://localhost:8000/v1
• Model — glm-5.2-colibri (or whatever you pass to --model-id)
• API key — any non-empty string, e.g. local

Colibri needs no API key by default, but many clients refuse to start without one — give them any dummy value. The key is only enforced if you set COLI_API_KEY.

Smoke-test the endpoint first (no key needed unless you set one):

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"glm-5.2-colibri","messages":[{"role":"user","content":"hi"}]}'

aider

export OPENAI_API_BASE=http://localhost:8000/v1
export OPENAI_API_KEY=local
aider --model openai/glm-5.2-colibri     # the openai/ prefix routes to OPENAI_API_BASE

crush — add a provider to crush.json (~/.config/crush/crush.json, or %USERPROFILE%\AppData\Local\crush\crush.json on Windows):

{
  "$schema": "https://charm.land/crush.json",
  "providers": {
    "colibri": {
      "name": "Colibri",
      "type": "openai-compat",
      "base_url": "http://localhost:8000/v1/",
      "api_key": "local",
      "models": [
        { "name": "GLM-5.2 (Colibri)", "id": "glm-5.2-colibri",
          "context_window": 131072, "default_max_tokens": 1024 }
      ]
    }
  }
}

The "api_key": "local" dummy is what satisfies clients that demand a key. context_window is only the client's budget display — set it to whatever your KV configuration actually allows.

Continue, Cline / Roo, llm, the OpenAI SDKs, … — set the provider's base URL to http://localhost:8000/v1, the model to glm-5.2-colibri, and any dummy key (OPENAI_API_KEY / OPENAI_BASE_URL for env-based tools).

Set your expectations before connecting an agentic CLI. Two costs dominate, and the first one is invisible until you know it's there:

1. Prefill. Coding agents (crush, aider in repo-map mode, Cline, …) send a large system prompt plus tool definitions — often 10–20k tokens — before your first word. Prefill on the CPU-streaming path runs at a few tokens per second (it is attention-bound, see #153), so a 15k-token agent preamble is an hour of silent "thinking" before the first output token. The client looks hung; it isn't. Smoke-test with the tiny curl above first — if that answers in about a minute, the pipeline works and what you're paying for is prompt size.
2. Decode. Roughly 1 tok/s for a large model, so multi-turn agent loops (which re-pay the growing context every turn) compound the cost.

Practical guidance: single surgical asks with a short context work; iterative agent sessions against a disk-streaming 744B model do not resemble a hosted API and mostly won't be worth the wait. If your client lets you trim or disable its system preamble and tool catalog, do it.

Isolated KV contexts

coli serve --kv-slots N allocates up to 16 independent sequence contexts. Requests select one with the optional integer cache_slot field; ordinary OpenAI clients omit it and keep the original slot 0 behavior.

{
  "model": "glm-5.2-colibri",
  "messages": [{"role": "user", "content": "Continue this conversation"}],
  "cache_slot": 1
}

Each slot owns its token history, compressed MLA/DSA KV memory, MTP window, and crash-safe persistence file (.coli_kv, .coli_kv.1, ...). The engine matches each request's tokenized prompt against the slot's history and reuses the common KV prefix, so stateless HTTP turns keep their cache across requests and even across engine restarts. Use COLI_KV_SLOTS=N as the environment equivalent. Start small: at the default 4096-token context, every slot costs hundreds of MB.

Web dashboard

One command serves the OpenAI-compatible API and the web console on the same port, then opens your browser when the engine is ready:

cd web && npm install && npm run build   # once
./coli web --model <model-dir>

What you get:

• Chat with live metrics: a flashing token counter while generating, then tok/s, time-to-first-token, prompt→completion counts and queue wait;
• Runtime panel: your hardware (CPU, GPUs + VRAM, RAM, cores), the scheduler, and the live expert-tier bar — how many of the 19,456 experts sit in VRAM / RAM / disk right now;
• Brain: the whole model as a 76×256 cortex, one cell per expert. Colour = tier, brightness = routing heat, and the experts routed in each turn flash white and decay — you watch the model think. Hover any cell for its tier, heat and measured topic affinity;
• Atlas: the measured expert atlas as a 3-D galaxy (publish experts.json from tools/expert_atlas/analyze.py --web).

The dashboard talks to the engine over a small line protocol and plain JSON endpoints — nothing heavier than the engine itself. web/ is a pure OpenAI-API client (React + TypeScript) and also works against any other compatible endpoint; the terminal coli chat remains the first-class interface.

The layout is responsive down to phone widths, and the sidebar carries the full telemetry stack — hardware, scheduler, tier bar, per-turn time breakdown, tok/s trend and per-GPU expert counts: