Rodolfo Hansen
8395435322
convert_fp8_to_int4.py could ingest FP8-blockscale, bf16 and f32
checkpoints, but not NVFP4 — the format NVIDIA modelopt emits for
REAP-pruned GLM-5.2 (quant_algo=NVFP4). Those expert weights are U8
(two e2m1 nibbles/byte) with a per-16-block FP8 scale sidecar
(.weight_scale) plus a per-tensor F32 global (.weight_scale_2).
Adds dequant_nvfp4(): e2m1 LUT (verified 1:1 against
ml_dtypes.float4_e2m1fn), low-nibble=even / high-nibble=odd unpacking,
per-block scale repeat-interleaved over group_size=16, times the global.
classify() now consumes the .weight_scale/.weight_scale_2/.input_scale
sidecars, and dequant() routes U8 tensors with a scale sidecar to the
NVFP4 path (keys is now required, so a stray U8 can't fall through to it).
Existing FP8/bf16/f32 paths are untouched.
Guards against silent corruption rather than trusting the input:
* group_size is fixed at 16 and the block-scale column count is
asserted == ceil(I/16); the old code inferred it from the data
(I // ncols), which misaligns silently on a padded/swizzled scale
layout and hard-crashes on a non-multiple-of-16 I — after a multi-GB
shard download. A partial trailing block is handled by slicing to I.
* .weight_scale_2 is asserted < 1: modelopt stores the small global and
MULTIPLIES; llm-compressor/compressed-tensors stores the reciprocal
(>= 1) and DIVIDES. The two are dtype-identical, so without this a
compressed-tensors checkpoint would corrupt every tensor by ~gscale^2.
--selftest-nvfp4 (no network) asserts the 16-code LUT, an encode->dequant
round-trip to <1e-9, and a dequant->colibri-int4 requant bound (<0.30;
the inherent ~0.17 is informational, not a precision claim).
Claude-Session: https://claude.ai/code/session_01DS7oc65c5RdA9V99otRCwt
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Tools
These scripts support model preparation and offline engineering work. They are not runtime dependencies of the C engine.
convert_fp8_to_int4.py,download_glm52.py: model preparationmake_glm_oracle.py,make_glm_bench_model.py: deterministic fixturesbenchmark_cuda_fixture.py,eval_glm.py,fetch_benchmarks.py: benchmarksgen_unicode.py: tokenizer table generation
Run them from c/, for example:
python3 tools/convert_fp8_to_int4.py --selftest
python3 tools/make_glm_bench_model.py --output /tmp/colibri-bench