serve stage 2: per-request grammars end-to-end — response_format -> SUBMIT -> grammar-forced drafts in the multiplexed server

The OpenAI gateway previously 400'd every response_format; the mux engine path
ran with speculation disabled entirely. Now:

- openai_server.py: response_format {"type":"json_object"} (generic ws-tolerant
  JSON grammar), {"type":"json_schema"} (schema forwarded as-is, compiled
  engine-side by schema_gbnf.h), and a raw-GBNF {"type":"gbnf"} extension.
  Draft-source semantics throughout: a schema the engine cannot compile costs
  the speedup, never the request and never the output.
- SUBMIT protocol: optional 7th field gbytes; grammar text appended to the
  payload after the prompt. 6-field headers unchanged (back-compatible).
- Engine: per-slot GrDraft (grammar_setup_text/grammar_teardown split out of
  the env-driven setup); walkers fed on every emitted token. Grammar-forced
  drafting in run_serve_mux for greedy requests: a drafting slot leaves the
  shared batch for one forward and runs the proven single-sequence verify path
  (kv_bind + step_all) — the same primitives prefill already uses per
  submission — then rejoins; rejected drafts' KV entries are overwritten by the
  next forward exactly like the existing prefix-truncation path. Sampling
  requests never draft (verification under sampling needs rejection resampling;
  out of scope).

Tests: 7-field SUBMIT parse cases; response_format->grammar plumbing incl.
fail cases; test doubles updated; generic JSON grammar parse+walk validated
against grammar.h. make test-c green; python suite green except the known
environmental memory_available failure (#150 fixes it).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
FABIOTESS
2026-07-14 15:26:32 +01:00
parent d7211632b4
commit d7b855f43f
5 changed files with 188 additions and 52 deletions
+6
View File
@@ -44,6 +44,12 @@ static void test_submit_header(void)
assert(coli_submit_parse("SUBMIT 1 0 16777216 3 1 1", &sub));
assert(!coli_submit_parse("SUBMIT 1 0 16777217 3 1 1", &sub));
assert(!coli_submit_parse("SUBMIT 1 0 2 3 1 1 trailing", &sub));
/* optional 7th field: per-request grammar length (0 when absent) */
assert(coli_submit_parse("SUBMIT 42 3 17 64 0.7 0.95", &sub) && sub.gbytes == 0);
assert(coli_submit_parse("SUBMIT 42 3 17 64 0.7 0.95 512", &sub) && sub.gbytes == 512);
assert(coli_submit_parse("SUBMIT 42 3 17 64 0.7 0.95 1048576", &sub));
assert(!coli_submit_parse("SUBMIT 42 3 17 64 0.7 0.95 1048577", &sub));
assert(!coli_submit_parse("SUBMIT 42 3 17 64 0.7 0.95 512 extra", &sub));
}
int main(void)