From 8a1dccae3b1a4c82ea9173e3d1ea8c002f14e785 Mon Sep 17 00:00:00 2001 From: JustVugg Date: Sat, 18 Jul 2026 02:33:23 +0200 Subject: [PATCH] docs(api): tell coding-CLI users about the prefill cost before they hit it (#373) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit yurivict connected crush per the walkthrough and got 'thought for 1h8m and did nothing' — which is not a hang: agent CLIs send a 10-20k-token system preamble, and prefill on the CPU-streaming path runs at a few tok/s (attention-bound, #153). An hour of silent prefill looks exactly like a dead server. The note now spells out the arithmetic, the curl smoke-test that separates slow-but-working from broken, and honest guidance on what agent workloads are (not) viable. Co-Authored-By: Claude Opus 4.8 --- docs/api.md | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-) diff --git a/docs/api.md b/docs/api.md index cb29eb2..62010bd 100644 --- a/docs/api.md +++ b/docs/api.md @@ -106,9 +106,24 @@ KV configuration actually allows. URL to `http://localhost:8000/v1`, the model to `glm-5.2-colibri`, and any dummy key (`OPENAI_API_KEY` / `OPENAI_BASE_URL` for env-based tools). -> On the CPU-streaming path a large model decodes at roughly 1 tok/s, so -> interactive agent loops will feel slow — it connects and works, but the latency -> is very different from a hosted model. +> **Set your expectations before connecting an agentic CLI.** Two costs dominate, +> and the first one is invisible until you know it's there: +> +> 1. **Prefill.** Coding agents (crush, aider in repo-map mode, Cline, …) send a +> large system prompt plus tool definitions — often 10–20k tokens — *before +> your first word*. Prefill on the CPU-streaming path runs at a few tokens per +> second (it is attention-bound, see #153), so a 15k-token agent preamble is +> **an hour of silent "thinking" before the first output token**. The client +> looks hung; it isn't. Smoke-test with the tiny `curl` above first — if that +> answers in about a minute, the pipeline works and what you're paying for is +> prompt size. +> 2. **Decode.** Roughly 1 tok/s for a large model, so multi-turn agent loops +> (which re-pay the growing context every turn) compound the cost. +> +> Practical guidance: single surgical asks with a short context work; iterative +> agent sessions against a disk-streaming 744B model do not resemble a hosted +> API and mostly won't be worth the wait. If your client lets you trim or disable +> its system preamble and tool catalog, do it. ## Isolated KV contexts