diskio: KV write batching + persistent handle; generators: unfuse experts

Two independent fixes validated end-to-end on fresh fixtures:

1. KV cache disk I/O (issue_diskio.md opportunities #1 + #4):
   - kv_disk_append: fopen/fclose every turn -> persistent FILE* kept open
     for the engine lifetime, lazy open on first append, closed in
     serve_ctx_free. Eliminates per-turn handle creation overhead.
   - kv_disk_append: ~157 small fwrites per position -> one contiguous
     record memcpy'd into a staging buffer then a single fwrite per
     position. The staging buffer grows on demand via realloc.
   - kv_disk_truncate: closes the persistent handle before truncating
     so the file actually shrinks on disc, then reopens lazily.
   - KVState gains disk_fp, disk_buf, disk_buf_cap fields.
   - Verified: serve-mode round-trip, write 11 tokens then reload and
     resume with no re-prefill, then append 8 more and reload to 19.

2. Expert weight unfusing in test-model generators:
   - The real GLM-5.2-FP8 checkpoint stores routed experts UNFUSED as
     per-expert 2-D tensors, each with its own _scale_inv. HF fuses
     gate+up into a single 3-D gate_up_proj for compute efficiency.
   - The converter and C engine both expect the unfused layout. The
     fused 3-D tensors were silently skipped by the converter, and the
     engine crashed with missing-tensor errors.
   - New unfuse_experts in glm_fp8_emit.py splits gate_up_proj and
     down_proj into per-expert 2-D tensors. Called after reference
     generation but before saving, in both generators, both FP8 and bf16.
   - Also fixed: make_glm_oracle.py FP8 round-trip guard used p.dim()<2
     which let 3-D fused experts through and crashed fp8_block_quantize.
     Changed to p.dim()!=2 to match the converter ndim!=2 guard.

Validated full chain on fresh fixtures:
  generator --fp8 -> 570 e4m3 tensors + 629 scale_inv, was 90 when fused
  converter --group-size 0  -> per-row int4 fmt=2, engine loads clean
  converter --group-size 128 -> grouped int4 fmt=4, 8-16x more scales,
    engine loads clean, fmt=4 auto-detected in both mmap and slab paths
  dequant error: grouped 1.14-1.22x lower than per-row vs FP8 source
This commit is contained in:
woolcoxm
2026-07-15 01:40:02 -04:00
parent e71d4fbe29
commit 71c262ce1a
4 changed files with 127 additions and 27 deletions
+42
View File
@@ -66,6 +66,48 @@ def fp8_block_dequantize(w_fp8, scale):
return qf * scale.repeat_interleave(BLOCK, 0).repeat_interleave(BLOCK, 1)[:O, :I]
def unfuse_experts(sd):
"""Split HF's fused 3-D `experts.gate_up_proj` [E, 2*M, I] into per-expert 2-D
`experts.{e}.gate_proj` [M, I] + `experts.{e}.up_proj` [M, I], and
`experts.down_proj` [E, I, M] -> `experts.{e}.down_proj` [M_out, I].
The real GLM-5.2-FP8 checkpoint stores experts UNFUSED as per-expert 2-D tensors
(gate_proj, up_proj, down_proj), each with its own _scale_inv. HF's
GlmMoeDsaForCausalLM fuses gate+up into a single 3-D gate_up_proj for efficiency.
The converter (classify + ndim!=2 guard) and the C engine both expect the unfused
layout, so we split before saving.
Idempotent: if experts are already unfused (no 3-D gate_up_proj), returns sd as-is.
EN: split HF's fused 3-D expert weights into the per-expert 2-D layout that the real
EN: checkpoint uses and the converter/engine expect. No-op if already unfused."""
keys_to_remove = []
new_entries = {}
for name, t in sd.items():
if not name.endswith(".mlp.experts.gate_up_proj"):
continue
# prefix = everything before ".mlp.experts.gate_up_proj"
prefix = name[:-len(".mlp.experts.gate_up_proj")]
E, twoM, I = t.shape # [E, 2*intermediate, input]
M = twoM // 2
for e in range(E):
new_entries[f"{prefix}.mlp.experts.{e}.gate_proj.weight"] = t[e, :M, :].contiguous()
new_entries[f"{prefix}.mlp.experts.{e}.up_proj.weight"] = t[e, M:, :].contiguous()
keys_to_remove.append(name)
# down_proj may be 3-D [E, I, M] in the fused form, or already per-expert
for name, t in sd.items():
if not name.endswith(".mlp.experts.down_proj") or t.dim() != 3:
continue
prefix = name[:-len(".mlp.experts.down_proj")]
E = t.shape[0]
for e in range(E):
new_entries[f"{prefix}.mlp.experts.{e}.down_proj.weight"] = t[e].contiguous()
keys_to_remove.append(name)
for k in keys_to_remove:
sd.pop(k, None)
sd.update(new_entries)
return sd
def state_dict_to_fp8(sd):
"""Converte uno state_dict HuggingFace nel layout FP8 del checkpoint reale:
per ogni tensore quantizzabile 2-D scrive `{name}` (F8_E4M3) + `{name}_scale_inv` (F32);