* prefill: batch the attention input projections (q_a/q_b/kv_a) over the prompt
During prefill the three attention input projections were called one row at a time
(matmul_qt(..., 1)) inside the loop over the S prompt tokens, while o_proj right
below already runs batched and moe() already does a batch-union. Batching them over
all S rows lets each weight row be read once for the whole prompt instead of once
per token.
Measured on Zen5 + RTX 5090, GLM-5.2 int4, 78 layers, DSA on, OMP tuning on,
prefill run to completion (engine's own timings, upstream/dev @ 6d3ed7e):
prompt projection/RoPE total prefill
256 22.7s -> 17.7s (-22%) 120.9s -> 115.4s (-4.5%)
512 45.5s -> 36.3s (-20%) 231.4s -> 222.7s (-3.8%)
Output is BIT-IDENTICAL to dev: summed log-lik of a fixed 1023-token passage is
-4987.458316 before and after (SCORE mode, CPU, deterministic). Decode (S=1) is
untouched — same calls, same kernel.
matmul_qt_ex(..., allow_idot=0) keeps the projections on the EXACT int4 kernel.
This matters: batching alone would push S past the S>=g_i4s gate (2 on x86) and
silently move them onto IDOT's int8-quantized activations, which costs
+0.169 nats/token on markdown/code and +0.084 on prose (perplexity +18.4% / +8.7%,
measured) for only 1.4% more speed now that #95 made the exact kernel fast. The
gate is a decode-era speed threshold; it should not be crossed by a batching change.
Also: matmul_qt documents the IDOT threshold as "configurable via I4S", but the
getenv was missing and the knob did nothing. Wire it up — it is what isolates the
kernel switch from the batching win.
* prefill: batch the DSA indexer's ix_wk projection too
Same per-token bug as the attention projections, one block below: ix_wk was called
one row at a time inside the prefill loop, so its weights were re-read for every
token on every DSA layer.
matmul_qt_ex(..., 0) keeps it on the exact int4 kernel for the same reason as the
attention projections: batching alone would push S past the S>=g_i4s gate and
silently change the activation quantization.
Output stays BIT-IDENTICAL (summed log-lik over 1023 tokens, in-distribution):
prose -2295.245383 (unchanged)
markdown -3272.146403 (unchanged)
Speed, 256-token prompt, 3 alternating reps each (Zen5 + RTX 5090, dev + the
attention-projection commit as the baseline):
projection/RoPE 17.54s (sd 0.20) -> 16.96s (sd 0.15) -3.3%
total prefill 115.34s (sd 0.82) -> 113.69s (sd 0.35) -1.4%
Both separated beyond 2 sd. Small, but it is free and bit-identical.
ix_wq/ix_wp are left alone on purpose: they sit inside an `omp parallel for` and
are skipped entirely below index_topk context, so batching them is a different
change with a different risk profile.
---------
Co-authored-by: JustVugg <JustVugg@users.noreply.github.com>
Dennis Paul
12350a494f