The tool tokenized every context with add_special_tokens=False, so pointing it at a
GLM snapshot scored the model out-of-distribution — the same bug @bokiko found in
eval_glm.py (#108).
Measured on this box (GLM-5.2 int4, engine SCORE mode, same 1023 target tokens,
only the conditioning prefix changed):
corpus no prefix +[gMASK]<sop>
natural prose PPL 29.2 PPL 9.4
markdown + code PPL 131.0 PPL 24.5
It does not just depress scores, it distorts sensitivity to numerical changes: an
exact-vs-IDOT kernel A/B measured without the prefix reported a penalty that halved
AND flipped sign on one of two corpora once the prefix was restored (#153). A
quantization-ablation tool that scores OOD is therefore worse than useless — it
produces confident deltas that are artifacts.
The GLM tokenizer does not add the prefix itself (add_special_tokens=True is a no-op
there), so it must be prepended explicitly. Auto-detected from the vocab rather than
opt-in, so it cannot be lost by omission; --prefix overrides. Models with no such
prefix are unaffected: OLMoE (the tool's default) has no BOS at all, add_special_tokens
True/False give identical ids, so the numbers in #108 measured on OLMoE still stand.
Dennis Paul
540781528e