fix(engine): filter non-EOS stop tokens in serve mode to prevent premature halt on tool calls
GLM-5.2's config defines three stop tokens: <|endoftext|>, <|user|>, and <|observation|>. In serve mode, when the model generates <tool_call> blocks, int4-quantized logit noise can cause argmax to pick a <|user|> or <|observation|> token ID, immediately stopping generation. The <|user|> and <|observation|> tokens are role markers handled by the Python API server, not the C engine. Filter them out in stops_arm() when SERVE=1, keeping only <|endoftext|> as the stop token. - Add SERVE-mode guard in stops_arm() that retains only the EOS token - Log the number of filtered tokens for diagnostics
This commit is contained in:
@@ -4115,6 +4115,19 @@ static void stops_arm(const Cfg *c, int tok_eos){
|
||||
g_nstop=0;
|
||||
for(int i=0;i<c->n_stop;i++) g_stop[g_nstop++]=c->stop_ids[i];
|
||||
if(tok_eos>=0 && !is_stop(tok_eos)) g_stop[g_nstop++]=tok_eos;
|
||||
/* In serve mode (API), keep only <|endoftext|> as a stop token. The tokens
|
||||
* <|user|> and <|observation|> are role markers that the Python server
|
||||
* handles — keeping them as stop tokens causes the engine to halt
|
||||
* prematurely when the model tries to emit <tool_call> blocks, because
|
||||
* int4-quantized logits can be noisy enough that argmax picks a stop-token
|
||||
* ID instead of the correct tool-call token. (#401) */
|
||||
if(getenv("SERVE")){
|
||||
int kept=0;
|
||||
for(int i=0;i<g_nstop;i++) if(g_stop[i]==tok_eos) g_stop[kept++]=g_stop[i];
|
||||
if(kept<g_nstop) fprintf(stderr,"[stop] serve mode: filtered %d non-EOS stop tokens (tool-call safety, #401)\n",
|
||||
g_nstop-kept);
|
||||
g_nstop=kept;
|
||||
}
|
||||
fprintf(stderr,"[stop] %d stop tokens:",g_nstop);
|
||||
for(int i=0;i<g_nstop;i++) fprintf(stderr," %d",g_stop[i]);
|
||||
fprintf(stderr,"\n");
|
||||
|
||||
Reference in New Issue
Block a user