docs(api): add "Connect a coding CLI or editor" recipe (#373)
Documents connecting OpenAI-compatible coding CLIs to `coli serve`. Covers the common snag reported in #373 — clients like crush refuse to start without an API key even though the local endpoint needs none — by showing that any dummy key works (Colibri only enforces COLI_API_KEY if set). Concrete recipes for aider and crush, a curl smoke test, the generic base-URL/model/key pattern for other tools, and an honest tok/s-latency caveat for the streaming path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
+61
@@ -49,6 +49,67 @@ errors before streaming headers are sent. `GET /health` exposes
|
|||||||
active/queued/completed/rejected counters, and successful generation responses
|
active/queued/completed/rejected counters, and successful generation responses
|
||||||
include `x-colibri-queue-wait-ms`.
|
include `x-colibri-queue-wait-ms`.
|
||||||
|
|
||||||
|
## Connect a coding CLI or editor
|
||||||
|
|
||||||
|
The API is OpenAI-compatible, so most coding CLIs and editor extensions work by
|
||||||
|
pointing them at Colibri as an *OpenAI-compatible* provider. Three settings:
|
||||||
|
|
||||||
|
- **Base URL** — `http://localhost:8000/v1`
|
||||||
|
- **Model** — `glm-5.2-colibri` (or whatever you pass to `--model-id`)
|
||||||
|
- **API key** — any non-empty string, e.g. `local`
|
||||||
|
|
||||||
|
Colibri needs **no** API key by default, but many clients refuse to start without
|
||||||
|
one — give them any dummy value. The key is only enforced if you set `COLI_API_KEY`.
|
||||||
|
|
||||||
|
Smoke-test the endpoint first (no key needed unless you set one):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl http://127.0.0.1:8000/v1/chat/completions \
|
||||||
|
-H 'Content-Type: application/json' \
|
||||||
|
-d '{"model":"glm-5.2-colibri","messages":[{"role":"user","content":"hi"}]}'
|
||||||
|
```
|
||||||
|
|
||||||
|
**aider**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export OPENAI_API_BASE=http://localhost:8000/v1
|
||||||
|
export OPENAI_API_KEY=local
|
||||||
|
aider --model openai/glm-5.2-colibri # the openai/ prefix routes to OPENAI_API_BASE
|
||||||
|
```
|
||||||
|
|
||||||
|
**crush** — add a provider to `crush.json` (`~/.config/crush/crush.json`, or
|
||||||
|
`%USERPROFILE%\AppData\Local\crush\crush.json` on Windows):
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"$schema": "https://charm.land/crush.json",
|
||||||
|
"providers": {
|
||||||
|
"colibri": {
|
||||||
|
"name": "Colibri",
|
||||||
|
"type": "openai-compat",
|
||||||
|
"base_url": "http://localhost:8000/v1/",
|
||||||
|
"api_key": "local",
|
||||||
|
"models": [
|
||||||
|
{ "name": "GLM-5.2 (Colibri)", "id": "glm-5.2-colibri",
|
||||||
|
"context_window": 131072, "default_max_tokens": 1024 }
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
The `"api_key": "local"` dummy is what satisfies clients that demand a key.
|
||||||
|
`context_window` is only the client's budget display — set it to whatever your
|
||||||
|
KV configuration actually allows.
|
||||||
|
|
||||||
|
**Continue, Cline / Roo, `llm`, the OpenAI SDKs, …** — set the provider's base
|
||||||
|
URL to `http://localhost:8000/v1`, the model to `glm-5.2-colibri`, and any dummy
|
||||||
|
key (`OPENAI_API_KEY` / `OPENAI_BASE_URL` for env-based tools).
|
||||||
|
|
||||||
|
> On the CPU-streaming path a large model decodes at roughly 1 tok/s, so
|
||||||
|
> interactive agent loops will feel slow — it connects and works, but the latency
|
||||||
|
> is very different from a hosted model.
|
||||||
|
|
||||||
## Isolated KV contexts
|
## Isolated KV contexts
|
||||||
|
|
||||||
`coli serve --kv-slots N` allocates up to 16 independent sequence contexts.
|
`coli serve --kv-slots N` allocates up to 16 independent sequence contexts.
|
||||||
|
|||||||
Reference in New Issue
Block a user