url/colibri
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

serve stage 2: per-request grammars end-to-end — response_format -> SUBMIT -> grammar-forced drafts in the multiplexed server

Browse Source 
The OpenAI gateway previously 400'd every response_format; the mux engine path
ran with speculation disabled entirely. Now:

- openai_server.py: response_format {"type":"json_object"} (generic ws-tolerant
  JSON grammar), {"type":"json_schema"} (schema forwarded as-is, compiled
  engine-side by schema_gbnf.h), and a raw-GBNF {"type":"gbnf"} extension.
  Draft-source semantics throughout: a schema the engine cannot compile costs
  the speedup, never the request and never the output.
- SUBMIT protocol: optional 7th field gbytes; grammar text appended to the
  payload after the prompt. 6-field headers unchanged (back-compatible).
- Engine: per-slot GrDraft (grammar_setup_text/grammar_teardown split out of
  the env-driven setup); walkers fed on every emitted token. Grammar-forced
  drafting in run_serve_mux for greedy requests: a drafting slot leaves the
  shared batch for one forward and runs the proven single-sequence verify path
  (kv_bind + step_all) — the same primitives prefill already uses per
  submission — then rejoins; rejected drafts' KV entries are overwritten by the
  next forward exactly like the existing prefix-truncation path. Sampling
  requests never draft (verification under sampling needs rejection resampling;
  out of scope).

Tests: 7-field SUBMIT parse cases; response_format->grammar plumbing incl.
fail cases; test doubles updated; generic JSON grammar parse+walk validated
against grammar.h. make test-c green; python suite green except the known
environmental memory_available failure (#150 fixes it).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
FABIOTESS
2026-07-14 15:26:32 +01:00
parent d7211632b4
commit d7b855f43f
5 changed files with 188 additions and 52 deletions
Download Patch File Download Diff File Expand all files Collapse all files
c/decode_batch.h
+17 -7
View File
@@ -13,22 +13,32 @@ static inline float *coli_kv_row(float *base, int position, int width)
}
typedef struct {
unsigned long long id, bytes;
unsigned long long id, bytes, gbytes;
int slot, max_tokens;
float temperature, top_p;
} ColiSubmit;
/* Parse the textual header. The payload is read separately using `bytes`, so
* it may contain newlines. Reject trailing fields to keep framing unambiguous. */
* it may contain newlines. Reject trailing fields to keep framing unambiguous.
* Optional 7th field `gbytes`: length of a per-request grammar (raw GBNF, or a
* JSON-Schema compiled engine-side) appended to the payload AFTER the prompt
* bytes. 6-field headers remain valid (gbytes = 0). */
static inline int coli_submit_parse(const char *line, ColiSubmit *s)
{
char tail;
if (!line || !s ||
sscanf(line, "SUBMIT %llu %d %llu %d %f %f %c", &s->id, &s->slot,
if (!line || !s) return 0;
s->gbytes = 0;
if (sscanf(line, "SUBMIT %llu %d %llu %d %f %f %llu %c", &s->id, &s->slot,
&s->bytes, &s->max_tokens, &s->temperature, &s->top_p,
&tail) != 6)
return 0;
return s->id > 0 && s->bytes <= (16u << 20) && s->slot >= 0 && s->max_tokens >= 1 &&
&s->gbytes, &tail) != 7) {
s->gbytes = 0;
if (sscanf(line, "SUBMIT %llu %d %llu %d %f %f %c", &s->id, &s->slot,
&s->bytes, &s->max_tokens, &s->temperature, &s->top_p,
&tail) != 6)
return 0;
}
return s->id > 0 && s->bytes <= (16u << 20) && s->gbytes <= (1u << 20) &&
s->slot >= 0 && s->max_tokens >= 1 &&
isfinite(s->temperature) && isfinite(s->top_p) &&
s->temperature >= 0 && s->temperature <= 2 &&
s->top_p > 0 && s->top_p <= 1;