FABIOTESS
5e42e70ec3
New weight format: int3 with ONE f32 scale per 64-input group (3.5 bits/weight effective). Per group: 16B low plane (2 bits/val, int2 layout) + 8B high plane (1 bit/val), values [-4,3] stored v+4. Same quantization math as tools/quant_ablation.py _quant_last_dim(bits=3, group=64) from #132, whose OLMoE ablation measured int3-g64 BEATING the shipped per-row int4 on quality (-7.5 vs -9.3pp) at ~14% fewer bits. Engine (placed per the #391 split): matmul_i3 + pack_int3_g64 + I3_* layout helpers in quant.h next to their kernel family; fmt=5 branches in colibri.c's qt_bytes/qt_alloc/qt_fill/matmul_qt/embed_row/qt_addrow/qt_matvec_rows. Format detection now goes through #413's qt_resolve_fmt: fmt=5 registers its distinct weight-byte layout O*ceil(I/64)*24 and its scale cardinality O*ceil(I/64) there, validated against [O,I] like every other format. int3-g64 and grouped-int4-at-gs=64 carry the SAME scale count, so the weight bytes are the int3 tag; row formats keep precedence for the small-I shapes where byte counts coincide. The io_uring expert path still used the raw ?1:?2:3 byte inference (it missed fmt=4 grouping entirely and never set gs) — converted to qt_resolve_fmt like the other two expert paths. Backends: qt_cuda_upload returns 0 for fmt=5 (tensor stays CPU-side, the documented fallback), the dense CUDA matmul gate excludes fmt=5, and Metal's existing fmt gates (gemm fmt<=3, moe fmt 1/2) already reject it. Converter: quant_int3_g64 in convert_fp8_to_int4.py; --ebits 3/--xbits 3 now emit it (previously bits=3 silently produced int4). Tests: tests/test_int3.c (bit-exact pack/unpack vs reference, matmul_i3 vs dequant-matmul incl. short tail groups and the real GLM I=7168, QT plumbing, qt_resolve_fmt disambiguation incl. the same-scale-count fmt=4/fmt=5 pair, outlier-rows RMS: int3-g64 3.3x lower error than per-row int4), tests/test_int3_load.c (hand-rolled .safetensors fixture through st_init + qt_from_disk: fmt=5 detected and loaded next to an int4 control tensor), tests/test_int3_convert.py (NumPy pack round-trip vs independent decoder). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Tools
These scripts support model preparation and offline engineering work. They are not runtime dependencies of the C engine.
convert_fp8_to_int4.py,download_glm52.py: model preparationmake_glm_oracle.py,make_glm_bench_model.py: deterministic fixturesbenchmark_cuda_fixture.py,eval_glm.py,fetch_benchmarks.py: benchmarksgen_unicode.py: tokenizer table generation
Run them from c/, for example:
python3 tools/convert_fp8_to_int4.py --selftest
python3 tools/make_glm_bench_model.py --output /tmp/colibri-bench