diskio: KV write batching + persistent handle; generators: unfuse experts
Two independent fixes validated end-to-end on fresh fixtures: 1. KV cache disk I/O (issue_diskio.md opportunities #1 + #4): - kv_disk_append: fopen/fclose every turn -> persistent FILE* kept open for the engine lifetime, lazy open on first append, closed in serve_ctx_free. Eliminates per-turn handle creation overhead. - kv_disk_append: ~157 small fwrites per position -> one contiguous record memcpy'd into a staging buffer then a single fwrite per position. The staging buffer grows on demand via realloc. - kv_disk_truncate: closes the persistent handle before truncating so the file actually shrinks on disc, then reopens lazily. - KVState gains disk_fp, disk_buf, disk_buf_cap fields. - Verified: serve-mode round-trip, write 11 tokens then reload and resume with no re-prefill, then append 8 more and reload to 19. 2. Expert weight unfusing in test-model generators: - The real GLM-5.2-FP8 checkpoint stores routed experts UNFUSED as per-expert 2-D tensors, each with its own _scale_inv. HF fuses gate+up into a single 3-D gate_up_proj for compute efficiency. - The converter and C engine both expect the unfused layout. The fused 3-D tensors were silently skipped by the converter, and the engine crashed with missing-tensor errors. - New unfuse_experts in glm_fp8_emit.py splits gate_up_proj and down_proj into per-expert 2-D tensors. Called after reference generation but before saving, in both generators, both FP8 and bf16. - Also fixed: make_glm_oracle.py FP8 round-trip guard used p.dim()<2 which let 3-D fused experts through and crashed fp8_block_quantize. Changed to p.dim()!=2 to match the converter ndim!=2 guard. Validated full chain on fresh fixtures: generator --fp8 -> 570 e4m3 tensors + 629 scale_inv, was 90 when fused converter --group-size 0 -> per-row int4 fmt=2, engine loads clean converter --group-size 128 -> grouped int4 fmt=4, 8-16x more scales, engine loads clean, fmt=4 auto-detected in both mmap and slab paths dequant error: grouped 1.14-1.22x lower than per-row vs FP8 source
This commit is contained in:
@@ -66,6 +66,48 @@ def fp8_block_dequantize(w_fp8, scale):
|
||||
return qf * scale.repeat_interleave(BLOCK, 0).repeat_interleave(BLOCK, 1)[:O, :I]
|
||||
|
||||
|
||||
def unfuse_experts(sd):
|
||||
"""Split HF's fused 3-D `experts.gate_up_proj` [E, 2*M, I] into per-expert 2-D
|
||||
`experts.{e}.gate_proj` [M, I] + `experts.{e}.up_proj` [M, I], and
|
||||
`experts.down_proj` [E, I, M] -> `experts.{e}.down_proj` [M_out, I].
|
||||
|
||||
The real GLM-5.2-FP8 checkpoint stores experts UNFUSED as per-expert 2-D tensors
|
||||
(gate_proj, up_proj, down_proj), each with its own _scale_inv. HF's
|
||||
GlmMoeDsaForCausalLM fuses gate+up into a single 3-D gate_up_proj for efficiency.
|
||||
The converter (classify + ndim!=2 guard) and the C engine both expect the unfused
|
||||
layout, so we split before saving.
|
||||
|
||||
Idempotent: if experts are already unfused (no 3-D gate_up_proj), returns sd as-is.
|
||||
EN: split HF's fused 3-D expert weights into the per-expert 2-D layout that the real
|
||||
EN: checkpoint uses and the converter/engine expect. No-op if already unfused."""
|
||||
keys_to_remove = []
|
||||
new_entries = {}
|
||||
for name, t in sd.items():
|
||||
if not name.endswith(".mlp.experts.gate_up_proj"):
|
||||
continue
|
||||
# prefix = everything before ".mlp.experts.gate_up_proj"
|
||||
prefix = name[:-len(".mlp.experts.gate_up_proj")]
|
||||
E, twoM, I = t.shape # [E, 2*intermediate, input]
|
||||
M = twoM // 2
|
||||
for e in range(E):
|
||||
new_entries[f"{prefix}.mlp.experts.{e}.gate_proj.weight"] = t[e, :M, :].contiguous()
|
||||
new_entries[f"{prefix}.mlp.experts.{e}.up_proj.weight"] = t[e, M:, :].contiguous()
|
||||
keys_to_remove.append(name)
|
||||
# down_proj may be 3-D [E, I, M] in the fused form, or already per-expert
|
||||
for name, t in sd.items():
|
||||
if not name.endswith(".mlp.experts.down_proj") or t.dim() != 3:
|
||||
continue
|
||||
prefix = name[:-len(".mlp.experts.down_proj")]
|
||||
E = t.shape[0]
|
||||
for e in range(E):
|
||||
new_entries[f"{prefix}.mlp.experts.{e}.down_proj.weight"] = t[e].contiguous()
|
||||
keys_to_remove.append(name)
|
||||
for k in keys_to_remove:
|
||||
sd.pop(k, None)
|
||||
sd.update(new_entries)
|
||||
return sd
|
||||
|
||||
|
||||
def state_dict_to_fp8(sd):
|
||||
"""Converte uno state_dict HuggingFace nel layout FP8 del checkpoint reale:
|
||||
per ogni tensore quantizzabile 2-D scrive `{name}` (F8_E4M3) + `{name}_scale_inv` (F32);
|
||||
|
||||
Reference in New Issue
Block a user