woolcoxm
498ab0c20e
Root cause of gibberish output: the int4 quantization uses one F32 scale per output row (2048 scales for a 2048x6144 matrix). The FP8 source has 128x128 block scales — 48x finer. This destroys reasoning while keeping surface fluency. Changes: - Converter: quant_int4_grouped() with --group-size 128 arg. Same nibble packing, but one scale per group of 128 elements along the input dim. - Engine QT struct: added 'gs' field (group size, 0=per-row backward compat) - Engine qt_from_disk: auto-detects fmt=4 when scale array is O*ceil(I/128) elements instead of O. Old per-row models (fmt=2) work unchanged. - Engine matmul_i4_grouped(): AVX2 kernel that applies per-group scales. Accumulator resets at each group boundary: dot(x[grp],w[grp]) * scale[grp]. - Engine matmul_qt_ex: dispatches to grouped kernel for fmt=4 (always exact, no IDOT approximation since the point is quality) - Engine expert_load: both mmap and slab+pread paths detect fmt=4 from scale array size and set gs=128 - qt_bytes: fmt=4 reports correct memory including group scales Backward compatible: existing per-row int4 models work unchanged. The fused gate+up pair path (matmul_i4_pair) falls back to separate matmul_qt calls for fmt=4 — minor perf cost, correctness preserved.
Tools
These scripts support model preparation and offline engineering work. They are not runtime dependencies of the C engine.
convert_fp8_to_int4.py,download_glm52.py: model preparationmake_glm_oracle.py,make_glm_bench_model.py: deterministic fixturesbenchmark_cuda_fixture.py,eval_glm.py,fetch_benchmarks.py: benchmarksgen_unicode.py: tokenizer table generation
Run them from c/, for example:
python3 tools/convert_fp8_to_int4.py --selftest
python3 tools/make_glm_bench_model.py --output /tmp/colibri-bench