 woolcoxm
|
69f65d5173
|
Windows-dev: grouped quantization + mixed precision + expert budget + download tool
Consolidates all experiment branches into one Windows-dev branch:
1. Group-scaled int4 (fmt=4, gs=128) — glm.c
- QT struct: added gs field
- matmul_i4_grouped: AVX2 kernel, verified to 3e-08 vs f32
- Format detection: auto-detects from .qs scale array size
- expert_load: both mmap and slab+pread paths handle fmt=4
- qt_bytes: fmt=4 case added
2. Per-tensor-type mixed precision — convert_fp8_to_int4.py
- Split classify() into sh/o/kvb/attn/dmlp sub-types
- New args: --shared-bits, --o-bits, --kvb-bits, --attn-bits, --dmlp-bits
- Plan: shared expert + o_proj + kv_b_proj at int8, rest grouped int4
- Only +5.3 GB RAM vs +0 for pure int4
3. EXPERT_BUDGET (miss-aware) — glm.c
- Caps distinct experts per layer across batch-union
- Always keeps cache hits, only drops misses
- Up to 1.8x faster decode on low-RAM hosts
4. Two-step shared-expert prediction (PILOT_TWO) — glm.c
- la_predict kind==2 + pilot_prefetch integration
- +3.1% recall over baseline PILOT
5. FP8 download tool — download_fp8.py
- ModelScope + HuggingFace dual-source
- Parallel shard download with stall recovery
6. Tiny model generation — make_glm_oracle.py
- Generated and tested locally for pipeline validation
|
2026-07-15 02:32:12 -04:00 |
|