FABIOTESS
5e42e70ec3
New weight format: int3 with ONE f32 scale per 64-input group (3.5 bits/weight effective). Per group: 16B low plane (2 bits/val, int2 layout) + 8B high plane (1 bit/val), values [-4,3] stored v+4. Same quantization math as tools/quant_ablation.py _quant_last_dim(bits=3, group=64) from #132, whose OLMoE ablation measured int3-g64 BEATING the shipped per-row int4 on quality (-7.5 vs -9.3pp) at ~14% fewer bits. Engine (placed per the #391 split): matmul_i3 + pack_int3_g64 + I3_* layout helpers in quant.h next to their kernel family; fmt=5 branches in colibri.c's qt_bytes/qt_alloc/qt_fill/matmul_qt/embed_row/qt_addrow/qt_matvec_rows. Format detection now goes through #413's qt_resolve_fmt: fmt=5 registers its distinct weight-byte layout O*ceil(I/64)*24 and its scale cardinality O*ceil(I/64) there, validated against [O,I] like every other format. int3-g64 and grouped-int4-at-gs=64 carry the SAME scale count, so the weight bytes are the int3 tag; row formats keep precedence for the small-I shapes where byte counts coincide. The io_uring expert path still used the raw ?1:?2:3 byte inference (it missed fmt=4 grouping entirely and never set gs) — converted to qt_resolve_fmt like the other two expert paths. Backends: qt_cuda_upload returns 0 for fmt=5 (tensor stays CPU-side, the documented fallback), the dense CUDA matmul gate excludes fmt=5, and Metal's existing fmt gates (gemm fmt<=3, moe fmt 1/2) already reject it. Converter: quant_int3_g64 in convert_fp8_to_int4.py; --ebits 3/--xbits 3 now emit it (previously bits=3 silently produced int4). Tests: tests/test_int3.c (bit-exact pack/unpack vs reference, matmul_i3 vs dequant-matmul incl. short tail groups and the real GLM I=7168, QT plumbing, qt_resolve_fmt disambiguation incl. the same-scale-count fmt=4/fmt=5 pair, outlier-rows RMS: int3-g64 3.3x lower error than per-row int4), tests/test_int3_load.c (hand-rolled .safetensors fixture through st_init + qt_from_disk: fmt=5 detected and loaded next to an int4 control tensor), tests/test_int3_convert.py (NumPy pack round-trip vs independent decoder). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
71 lines
2.8 KiB
Python
71 lines
2.8 KiB
Python
"""quant_int3_g64 (tools/convert_fp8_to_int4.py): pack layout + round-trip.
|
|
|
|
Decodes the packed bytes with an independent NumPy decoder implementing the
|
|
fmt=5 spec (16B low plane / 8B high plane per 64-group, v+4, per-group f32
|
|
scale) and checks the dequantized result equals the reference
|
|
quantize-dequantize (same math as quant_ablation._quant_last_dim(3, 64)).
|
|
The C side of the same layout is covered by tests/test_int3.c.
|
|
"""
|
|
import os, sys, unittest
|
|
try:
|
|
import numpy as np
|
|
except ImportError:
|
|
raise unittest.SkipTest("numpy not installed")
|
|
|
|
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "tools"))
|
|
from convert_fp8_to_int4 import quant_int3_g64
|
|
|
|
|
|
def decode(packed, scales, O, I, group=64):
|
|
ng = (I + group - 1) // group
|
|
b = packed.reshape(O, ng, 24)
|
|
lo, hi = b[:, :, :16], b[:, :, 16:]
|
|
k = np.arange(group)
|
|
lov = (lo[:, :, k >> 2] >> ((k & 3) * 2)[None, None, :]) & 3
|
|
hiv = (hi[:, :, k >> 3] >> (k & 7)[None, None, :]) & 1
|
|
v = (lov | (hiv << 2)).astype(np.int64) - 4
|
|
dq = v.astype(np.float64) * scales.reshape(O, ng, 1).astype(np.float64)
|
|
return dq.reshape(O, ng * group)[:, :I]
|
|
|
|
|
|
def reference(w, group=64):
|
|
"""same math as quant_int3_g64 (which works in f32), replayed exactly, then
|
|
dequantized in f64 so it matches decode() bit for bit"""
|
|
O, I = w.shape
|
|
ng = (I + group - 1) // group
|
|
pad = ng * group - I
|
|
wp = np.pad(w, ((0, 0), (0, pad))) if pad else w
|
|
g = wp.reshape(O, ng, group)
|
|
s = np.maximum(np.abs(g).max(axis=2, keepdims=True) / 3.0, 1e-8).astype(np.float32)
|
|
q = np.clip(np.rint(g / s), -4, 3).astype(np.int64)
|
|
return (q.astype(np.float64) * s.astype(np.float64)).reshape(O, ng * group)[:, :I]
|
|
|
|
|
|
class Int3ConvertTest(unittest.TestCase):
|
|
def test_round_trip(self):
|
|
rng = np.random.default_rng(7)
|
|
for I in (64, 128, 100, 65, 7168):
|
|
w = (rng.standard_normal((5, I)) * 0.05).astype(np.float32)
|
|
w[0, 3] = 1.7; w[2, min(5, I - 1)] = -2.2
|
|
packed, scales = quant_int3_g64(w)
|
|
ng = (I + 63) // 64
|
|
self.assertEqual(packed.size, 5 * ng * 24)
|
|
self.assertEqual(scales.size, 5 * ng)
|
|
np.testing.assert_allclose(decode(packed, scales, 5, I),
|
|
reference(w), rtol=0, atol=0)
|
|
|
|
def test_outliers_beat_row_int4(self):
|
|
rng = np.random.default_rng(11)
|
|
w = (rng.standard_normal((8, 1024)) * 0.02).astype(np.float32)
|
|
for o in range(8): w[o, (o * 37) % 1024] = 1.5
|
|
packed, scales = quant_int3_g64(w)
|
|
e3 = float(((decode(packed, scales, 8, 1024) - w) ** 2).mean())
|
|
s4 = np.maximum(np.abs(w).max(axis=1, keepdims=True) / 7.0, 1e-8)
|
|
w4 = np.clip(np.rint(w / s4), -8, 7) * s4
|
|
e4 = float(((w4 - w) ** 2).mean())
|
|
self.assertLess(e3, e4)
|
|
|
|
|
|
if __name__ == "__main__":
|
|
unittest.main()
|