FABIOTESS
d7b855f43f
The OpenAI gateway previously 400'd every response_format; the mux engine path
ran with speculation disabled entirely. Now:
- openai_server.py: response_format {"type":"json_object"} (generic ws-tolerant
JSON grammar), {"type":"json_schema"} (schema forwarded as-is, compiled
engine-side by schema_gbnf.h), and a raw-GBNF {"type":"gbnf"} extension.
Draft-source semantics throughout: a schema the engine cannot compile costs
the speedup, never the request and never the output.
- SUBMIT protocol: optional 7th field gbytes; grammar text appended to the
payload after the prompt. 6-field headers unchanged (back-compatible).
- Engine: per-slot GrDraft (grammar_setup_text/grammar_teardown split out of
the env-driven setup); walkers fed on every emitted token. Grammar-forced
drafting in run_serve_mux for greedy requests: a drafting slot leaves the
shared batch for one forward and runs the proven single-sequence verify path
(kv_bind + step_all) — the same primitives prefill already uses per
submission — then rejoins; rejected drafts' KV entries are overwritten by the
next forward exactly like the existing prefix-truncation path. Sampling
requests never draft (verification under sampling needs rejection resampling;
out of scope).
Tests: 7-field SUBMIT parse cases; response_format->grammar plumbing incl.
fail cases; test doubles updated; generic JSON grammar parse+walk validated
against grammar.h. make test-c green; python suite green except the known
environmental memory_available failure (#150 fixes it).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
48 lines
1.7 KiB
C
48 lines
1.7 KiB
C
#ifndef COLIBRI_DECODE_BATCH_H
|
|
#define COLIBRI_DECODE_BATCH_H
|
|
|
|
#include <stddef.h>
|
|
#include <stdio.h>
|
|
#include <math.h>
|
|
|
|
/* `base` belongs to one sequence's KV state. Keeping this arithmetic in a
|
|
* model-independent seam makes ragged decode row ownership directly testable. */
|
|
static inline float *coli_kv_row(float *base, int position, int width)
|
|
{
|
|
return base + (size_t)position * (size_t)width;
|
|
}
|
|
|
|
typedef struct {
|
|
unsigned long long id, bytes, gbytes;
|
|
int slot, max_tokens;
|
|
float temperature, top_p;
|
|
} ColiSubmit;
|
|
|
|
/* Parse the textual header. The payload is read separately using `bytes`, so
|
|
* it may contain newlines. Reject trailing fields to keep framing unambiguous.
|
|
* Optional 7th field `gbytes`: length of a per-request grammar (raw GBNF, or a
|
|
* JSON-Schema compiled engine-side) appended to the payload AFTER the prompt
|
|
* bytes. 6-field headers remain valid (gbytes = 0). */
|
|
static inline int coli_submit_parse(const char *line, ColiSubmit *s)
|
|
{
|
|
char tail;
|
|
if (!line || !s) return 0;
|
|
s->gbytes = 0;
|
|
if (sscanf(line, "SUBMIT %llu %d %llu %d %f %f %llu %c", &s->id, &s->slot,
|
|
&s->bytes, &s->max_tokens, &s->temperature, &s->top_p,
|
|
&s->gbytes, &tail) != 7) {
|
|
s->gbytes = 0;
|
|
if (sscanf(line, "SUBMIT %llu %d %llu %d %f %f %c", &s->id, &s->slot,
|
|
&s->bytes, &s->max_tokens, &s->temperature, &s->top_p,
|
|
&tail) != 6)
|
|
return 0;
|
|
}
|
|
return s->id > 0 && s->bytes <= (16u << 20) && s->gbytes <= (1u << 20) &&
|
|
s->slot >= 0 && s->max_tokens >= 1 &&
|
|
isfinite(s->temperature) && isfinite(s->top_p) &&
|
|
s->temperature >= 0 && s->temperature <= 2 &&
|
|
s->top_p > 0 && s->top_p <= 1;
|
|
}
|
|
|
|
#endif
|