Per-row int4 quantization of the MTP head's eh_proj [D,2D] zeroes its ENTIRE
embedding half: the tensor's two column halves differ in scale by ~20-30x per
row (embedding-half absmax ~0.05, hidden-half ~1.5 on GLM-5.2), so a single
per-row scale (absmax/7) puts every embedding-half weight below half a
quantization step and np.rint rounds it to exact zero (packed bytes 0x88).
The draft head then cannot see the input token and MTP acceptance collapses
to ~0% — the mechanism behind issue #8's measurement (int4: 0-4%; int8:
39-59%), and the reason --mtp already defaults to --ebits 8.
This showed up in the wild: a published pre-converted container had its MTP
shards made with an explicit --ebits 4 and drafts were pure garbage on every
backend (deterministic, data-side; verified byte-for-byte by reproducing the
published bytes with quant_int4 on the official BF16 rows).
- tools/repair_mtp_int8.py: repairs such a container IN PLACE — finds the MTP
layer's per-row-int4 dense tensors (eh_proj, q/kv/o projections, shared
experts; routed experts untouched), re-downloads only those from the FP8
source repo (~355 MB of HTTP range reads, no torch, no token), requantizes
at int8 with quant_int8's exact math, and rewrites the shards atomically
with *.bak-int4 backups. --dry-run to inspect. The engine picks up int8
automatically (qt_from_disk detects format by blob size).
Validated on GLM-5.2 744B: MTP acceptance 0% -> 100% (greedy), 1.11 -> 3.20
tokens/forward, decode 0.18 -> 0.30 tok/s (Metal, 4-bit KV).
- convert_fp8_to_int4.py: print a warning when --mtp is combined with
--ebits <8 and per-row scales (the exact footgun above); --group-size 128
remains a valid int4 alternative since group scales give the embedding
half its own scale.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
recviking
d6676c10e9