ZacharyZcR
e48346162a
#453 settled the scheme; this produces the actual bytes. iq3_pack.encode/decode implement the container layout so the converter, the engine and the decode kernels can all be written against one spec: 98 bytes per 256 weights = 3.0625 bpw [ 0..63] uint8 grid index per 4-dim magnitude block [64..95] uint32 x8 — four 7-bit sign words + 4-bit sub-scale per 32 [96..97] fp16 super-scale Signs use the published odd-parity trick: 7 of every 8 are stored and the 8th is derived, so the encoder flips the smallest-magnitude weight of any block whose true signs would violate parity — the cost the #453 ablation priced in, now actually paid. Sub-scale is searched over all 16 codes against the stored (fp16-rounded) super-scale, so encode-time and decode-time arithmetic agree. GLM-5.2 routed experts under this container: 372.7 -> 281.2 GB (-24.6%), and a 176 GB VRAM tier holds ~12,180 experts instead of ~9,190 (+33%). tests/test_iq3_pack.py: byte budget, deterministic encode, decode checked value-by-value against an independent loop-based reader written straight from the layout, sign-parity closure, and reconstruction quality in the band the chosen scheme measured (rel-RMSE 0.195 vs the torch model's 0.195).
Tools
These scripts support model preparation and offline engineering work. They are not runtime dependencies of the C engine.
convert_fp8_to_int4.py,download_glm52.py: model preparationmake_glm_oracle.py,make_glm_bench_model.py: deterministic fixturesbenchmark_cuda_fixture.py,eval_glm.py,fetch_benchmarks.py: benchmarksgen_unicode.py: tokenizer table generation
Run them from c/, for example:
python3 tools/convert_fp8_to_int4.py --selftest
python3 tools/make_glm_bench_model.py --output /tmp/colibri-bench