fix(engine): filter non-EOS stop tokens in serve mode to prevent premature halt on tool calls

GLM-5.2's config defines three stop tokens: <|endoftext|>, <|user|>, and
<|observation|>. In serve mode, when the model generates <tool_call> blocks,
int4-quantized logit noise can cause argmax to pick a <|user|> or
<|observation|> token ID, immediately stopping generation.

The <|user|> and <|observation|> tokens are role markers handled by the
Python API server, not the C engine. Filter them out in stops_arm() when
SERVE=1, keeping only <|endoftext|> as the stop token.

- Add SERVE-mode guard in stops_arm() that retains only the EOS token
- Log the number of filtered tokens for diagnostics
This commit is contained in:
Colibri Developer
2026-07-19 13:39:11 -06:00
parent 70fb5b00f3
commit 26bd8b403a
+13
View File
@@ -4115,6 +4115,19 @@ static void stops_arm(const Cfg *c, int tok_eos){
g_nstop=0;
for(int i=0;i<c->n_stop;i++) g_stop[g_nstop++]=c->stop_ids[i];
if(tok_eos>=0 && !is_stop(tok_eos)) g_stop[g_nstop++]=tok_eos;
/* In serve mode (API), keep only <|endoftext|> as a stop token. The tokens
* <|user|> and <|observation|> are role markers that the Python server
* handles keeping them as stop tokens causes the engine to halt
* prematurely when the model tries to emit <tool_call> blocks, because
* int4-quantized logits can be noisy enough that argmax picks a stop-token
* ID instead of the correct tool-call token. (#401) */
if(getenv("SERVE")){
int kept=0;
for(int i=0;i<g_nstop;i++) if(g_stop[i]==tok_eos) g_stop[kept++]=g_stop[i];
if(kept<g_nstop) fprintf(stderr,"[stop] serve mode: filtered %d non-EOS stop tokens (tool-call safety, #401)\n",
g_nstop-kept);
g_nstop=kept;
}
fprintf(stderr,"[stop] %d stop tokens:",g_nstop);
for(int i=0;i<g_nstop;i++) fprintf(stderr," %d",g_stop[i]);
fprintf(stderr,"\n");