Routed experts (gate/up/down) can now take different bit widths via PROJ_BITS.
Motivating config: --xbits 4 --up-bits 3 puts up_proj at int3-g64 (fmt=5, this
PR's format) while gate/down stay int4 — ~8% fewer expert bytes on disk and per
token, at ~zero quality cost. Backed by the OLMoE per-projection ablation posted
to #168: up@int3 matches int4-g64 (56.2 vs 55.8), up@int2 craters (-16pp).
This also supplies the definition the #404 resume manifests already depend on:
current dev records dict(PROJ_BITS) in check_or_record_params and the --indir
progress file in four places, but the global was never defined — every one of
those paths NameErrors at runtime today. The manifests were written for this
interface; this commit is the other half.
Validated: synthetic GLM fixture with --up-bits 3 yields int3-g64 up_proj
(O*(I/64)*24B weight + group scales) and int4-per-row gate/down.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New weight format: int3 with ONE f32 scale per 64-input group (3.5 bits/weight
effective). Per group: 16B low plane (2 bits/val, int2 layout) + 8B high plane
(1 bit/val), values [-4,3] stored v+4. Same quantization math as
tools/quant_ablation.py _quant_last_dim(bits=3, group=64) from #132, whose
OLMoE ablation measured int3-g64 BEATING the shipped per-row int4 on quality
(-7.5 vs -9.3pp) at ~14% fewer bits.
Engine (placed per the #391 split): matmul_i3 + pack_int3_g64 + I3_* layout
helpers in quant.h next to their kernel family; fmt=5 branches in colibri.c's
qt_bytes/qt_alloc/qt_fill/matmul_qt/embed_row/qt_addrow/qt_matvec_rows.
Format detection now goes through #413's qt_resolve_fmt: fmt=5 registers its
distinct weight-byte layout O*ceil(I/64)*24 and its scale cardinality
O*ceil(I/64) there, validated against [O,I] like every other format. int3-g64
and grouped-int4-at-gs=64 carry the SAME scale count, so the weight bytes are
the int3 tag; row formats keep precedence for the small-I shapes where byte
counts coincide. The io_uring expert path still used the raw ?1:?2:3 byte
inference (it missed fmt=4 grouping entirely and never set gs) — converted to
qt_resolve_fmt like the other two expert paths.
Backends: qt_cuda_upload returns 0 for fmt=5 (tensor stays CPU-side, the
documented fallback), the dense CUDA matmul gate excludes fmt=5, and Metal's
existing fmt gates (gemm fmt<=3, moe fmt 1/2) already reject it.
Converter: quant_int3_g64 in convert_fp8_to_int4.py; --ebits 3/--xbits 3 now
emit it (previously bits=3 silently produced int4).
Tests: tests/test_int3.c (bit-exact pack/unpack vs reference, matmul_i3 vs
dequant-matmul incl. short tail groups and the real GLM I=7168, QT plumbing,
qt_resolve_fmt disambiguation incl. the same-scale-count fmt=4/fmt=5 pair,
outlier-rows RMS: int3-g64 3.3x lower error than per-row int4),
tests/test_int3_load.c (hand-rolled .safetensors fixture through st_init +
qt_from_disk: fmt=5 detected and loaded next to an int4 control tensor),
tests/test_int3_convert.py (NumPy pack round-trip vs independent decoder).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Upstream courtesy fix, found while auditing the conversion recipe for this
branch -- pre-existing in dev, not introduced by int3-g64, but directly
relevant to anyone converting for real (a #383-class gap: same failure
family as the resume/manifest work already done for --indir, and the
silent-mixing mechanism issue #355 fixed for a narrower case).
The --indir path already refuses to resume with different conversion
parameters on the same --outdir (a manifest records ebits/xbits/io_bits/
group_size/n_layers/bits_map and compares on every resume). The --repo
streaming download loops (main model, --mtp, --indexer) never got the
same guard: each shard's resume check is just `if os.path.exists(outp):
continue` -- true whether or not THIS run's flags match the flags that
produced that shard. A --repo conversion resumed with changed bits
(--xbits 3 -> 4 mid-run, say, after an interruption) would silently mix
bit-widths across shards in the same container, with no error and no log
line distinguishing it from a normal resume.
Fix: check_or_record_params(), a small shared helper mirroring the
--indir manifest's refuse-on-mismatch logic but without needing its
per-shard bookkeeping (the --repo loops already track shard completion
correctly via out-NNNNN.safetensors existence, since shard index maps
directly to output filename there -- only whether the params used SO FAR
still match needed adding). Applied to all three --repo loops with
per-mode sidecar files (.out-mtp-params.json / .out-idx-params.json /
.out-params.json) so a --mtp and a main-model conversion into the same
--outdir don't cross-check each other's parameters.
Also includes PROJ_BITS (the per-projection expert bit overrides) in the
tracked params dict on BOTH paths -- it was missing from --indir's
existing manifest too, so a resume with a changed --up-bits/--gate-bits/
--down-bits would have passed the existing guard silently.
Verified directly (no real HF downloads; --repo network paths can't be
exercised under this task's constraints): unit-tested
check_or_record_params() standalone -- fresh outdir accepts and records,
a same-params resume accepts, a changed --xbits is refused, and a
proj_bits-only change (nothing else different) is refused. Re-ran the
existing --indir dry-run end to end (convert, resume, resume-with-changed-
xbits) to confirm the manifest-based path still works correctly with
proj_bits added to its params dict.
Gates: make test-c (20/20) and make test-python (85/85) both pass.
Per-row int4 quantization of the MTP head's eh_proj [D,2D] zeroes its ENTIRE
embedding half: the tensor's two column halves differ in scale by ~20-30x per
row (embedding-half absmax ~0.05, hidden-half ~1.5 on GLM-5.2), so a single
per-row scale (absmax/7) puts every embedding-half weight below half a
quantization step and np.rint rounds it to exact zero (packed bytes 0x88).
The draft head then cannot see the input token and MTP acceptance collapses
to ~0% — the mechanism behind issue #8's measurement (int4: 0-4%; int8:
39-59%), and the reason --mtp already defaults to --ebits 8.
This showed up in the wild: a published pre-converted container had its MTP
shards made with an explicit --ebits 4 and drafts were pure garbage on every
backend (deterministic, data-side; verified byte-for-byte by reproducing the
published bytes with quant_int4 on the official BF16 rows).
- tools/repair_mtp_int8.py: repairs such a container IN PLACE — finds the MTP
layer's per-row-int4 dense tensors (eh_proj, q/kv/o projections, shared
experts; routed experts untouched), re-downloads only those from the FP8
source repo (~355 MB of HTTP range reads, no torch, no token), requantizes
at int8 with quant_int8's exact math, and rewrites the shards atomically
with *.bak-int4 backups. --dry-run to inspect. The engine picks up int8
automatically (qt_from_disk detects format by blob size).
Validated on GLM-5.2 744B: MTP acceptance 0% -> 100% (greedy), 1.11 -> 3.20
tokens/forward, decode 0.18 -> 0.30 tok/s (Metal, 4-bit KV).
- convert_fp8_to_int4.py: print a warning when --mtp is combined with
--ebits <8 and per-row scales (the exact footgun above); --group-size 128
remains a valid int4 alternative since group scales give the embedding
half its own scale.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two gaps in the local path, both raised in #383:
- The main --indir pass copied only config.json while the download path
copies four files; without tokenizer.json the converted container can't
run chat/serve. Copy the same four, print what was copied, and warn
when the source is missing any (tokenizer.json called out explicitly).
- Local passes restarted from shard 0 on every rerun. out-NNNNN names
count EMITTED shards, not input indexes (shards with no relevant
tensors emit nothing), so the download path's exists-check can't be
mirrored directly: a sidecar manifest per prefix records
input -> output (or empty) plus the conversion parameters, written
atomically after each shard. Rerun skips what matches and refuses a
parameter mismatch on the same outdir instead of mixing containers
(the #355 failure mode).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
When --indir contains model.safetensors.index.json, the head/indexer
passes convert only the shards that actually hold the requested tensors
(3 instead of scanning 141 for --mtp — every empty scan still opens a
5 GB shard). Prints the selection in the [PLAN] block. Without the index
the full scan is unchanged.
Output is byte-identical to the unfiltered path (sha256-verified on the
GLM-5.2 FP8 head shards, with a decoy shard proving the filter excludes
rather than reorders).
Co-Authored-By: Claude <noreply@anthropic.com>
Two operator-surprise fixes:
openai_server.py (#382, LordMZTE): a request without max_tokens defaulted to
min(256, limit) — so `coli serve --ngen 32768` still cut every answer at 256
tokens for clients that omit the field. The operator's configured budget IS the
default now; generation still ends at EOS, so it's a cap, not a target.
convert_fp8_to_int4.py (#383, bokiko): the resolved conversion plan (mode,
source, ebits/io/x bits, grouped-vs-per-row) prints as a [PLAN] line before any
work. The two traps in #383 — --mtp defaulting ebits to 8 with the grouped
branch silently gated off, and --mtp appearing to ignore --indir — are already
defused by the #355 fix (the --mtp pass emits ONLY head tensors on the local
path), but a 3.5-hour job must show its plan at second 1, not in a post-hoc
size sanity check. bokiko's exact invocation now prints:
[PLAN] mode: MTP head only | source: local ./fp8 | experts 8-bit, ... |
PER-ROW (grouped branch needs bits<=4; ebits=8 disables it)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The local (`--indir`) branch of main() ignored --mtp and --indexer entirely:
it always called convert_shard() without keep_mtp/keep_idx and always wrote
`out-NNNNN.safetensors`. So the documented two-pass workflow — convert the main
model into an outdir, then run `--mtp` into the SAME outdir for the head — did
the opposite on the local path: with --mtp defaulting ebits to 8 and keep_mtp
staying False, it silently re-converted the whole model to per-row int8 and
overwrote the finished fmt=4 container shard by shard, printing nothing wrong.
@mohamedmastouri2000-boop lost 137 of 141 freshly-converted g64 shards to this
while building the public container for #298/#326.
The --indir branch now mirrors the download path: keep_mtp=a.mtp /
keep_idx=a.indexer passed through, output named out-mtp-/out-idx-/out- by mode,
empty shards skipped (an MTP pass emits only shards containing layer n_layers),
and config/tokenizer copied only on the main pass (the head/idx passes land in
an already-complete outdir).
Verified with a synthetic 2-layer + MTP-shard model: after the main pass, a
second --mtp pass into the same outdir leaves out-00000's md5 BYTE-IDENTICAL
and writes out-mtp-00000 alongside it. The bug would have changed that md5.
Reported-by: mohamedmastouri2000-boop <#355>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Split the resident weight classification into 5 sub-types so each can
get different precision:
sh = shared expert (highest sensitivity, fires every token)
o = o_proj (reconstructs output, biggest attn tensor)
kvb = kv_b_proj (reconstructs KV cache on every decode)
attn = q_a/q_b/kv_a (other attention projections)
dmlp = dense MLP (first 3 layers)
New args: --shared-bits, --o-bits, --kvb-bits, --attn-bits, --dmlp-bits
Each defaults to ebits (backward compat). When set, the converter applies
that precision to just that tensor type.
Research-backed plan: put the 3 compounding tensors (shared expert, o_proj,
kv_b_proj) at int8 and everything else at grouped int4. Extra RAM cost:
only +5.3 GB (those tensors are small vs the 372 GB expert pool on disk).
Root cause of gibberish output: the int4 quantization uses one F32 scale per
output row (2048 scales for a 2048x6144 matrix). The FP8 source has 128x128
block scales — 48x finer. This destroys reasoning while keeping surface fluency.
Changes:
- Converter: quant_int4_grouped() with --group-size 128 arg. Same nibble
packing, but one scale per group of 128 elements along the input dim.
- Engine QT struct: added 'gs' field (group size, 0=per-row backward compat)
- Engine qt_from_disk: auto-detects fmt=4 when scale array is O*ceil(I/128)
elements instead of O. Old per-row models (fmt=2) work unchanged.
- Engine matmul_i4_grouped(): AVX2 kernel that applies per-group scales.
Accumulator resets at each group boundary: dot(x[grp],w[grp]) * scale[grp].
- Engine matmul_qt_ex: dispatches to grouped kernel for fmt=4 (always exact,
no IDOT approximation since the point is quality)
- Engine expert_load: both mmap and slab+pread paths detect fmt=4 from
scale array size and set gs=128
- qt_bytes: fmt=4 reports correct memory including group scales
Backward compatible: existing per-row int4 models work unchanged.
The fused gate+up pair path (matmul_i4_pair) falls back to separate
matmul_qt calls for fmt=4 — minor perf cost, correctness preserved.
convert_fp8_to_int4.py could ingest FP8-blockscale, bf16 and f32
checkpoints, but not NVFP4 — the format NVIDIA modelopt emits for
REAP-pruned GLM-5.2 (quant_algo=NVFP4). Those expert weights are U8
(two e2m1 nibbles/byte) with a per-16-block FP8 scale sidecar
(.weight_scale) plus a per-tensor F32 global (.weight_scale_2).
Adds dequant_nvfp4(): e2m1 LUT (verified 1:1 against
ml_dtypes.float4_e2m1fn), low-nibble=even / high-nibble=odd unpacking,
per-block scale repeat-interleaved over group_size=16, times the global.
classify() now consumes the .weight_scale/.weight_scale_2/.input_scale
sidecars, and dequant() routes U8 tensors with a scale sidecar to the
NVFP4 path (keys is now required, so a stray U8 can't fall through to it).
Existing FP8/bf16/f32 paths are untouched.
Guards against silent corruption rather than trusting the input:
* group_size is fixed at 16 and the block-scale column count is
asserted == ceil(I/16); the old code inferred it from the data
(I // ncols), which misaligns silently on a padded/swizzled scale
layout and hard-crashes on a non-multiple-of-16 I — after a multi-GB
shard download. A partial trailing block is handled by slicing to I.
* .weight_scale_2 is asserted < 1: modelopt stores the small global and
MULTIPLIES; llm-compressor/compressed-tensors stores the reciprocal
(>= 1) and DIVIDES. The two are dtype-identical, so without this a
compressed-tensors checkpoint would corrupt every tensor by ~gscale^2.
--selftest-nvfp4 (no network) asserts the 16-code LUT, an encode->dequant
round-trip to <1e-9, and a dequant->colibri-int4 requant bound (<0.30;
the inherent ~0.17 is informational, not a precision claim).
Claude-Session: https://claude.ai/code/session_01DS7oc65c5RdA9V99otRCwt
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Three crash bugs in convert_fp8_to_int4.py, all on the download path:
P3: 'import fcntl' at line 211 is Unix-only — ModuleNotFoundError on Windows.
The --indir test path returns before reaching it so tests pass, but
'--repo' on Windows hard-crashes. Guarded the import: try fcntl (Unix),
fall back to msvcrt.locking (Windows), skip if neither available.
P4: repo_info retry loop had range(999) — up to ~16 hours of retries on a
bad network, then fell through to line 395 where 'info' was unbound
(NameError). Capped at 10 retries and added an explicit error + return
when exhausted. Also added an early return if no safetensors shards are
found in the repo.
P5: if the shards list was empty (wrong repo, all filtered out), the
'for i, sh in enumerate(shards)' loop never executed and 'i' was unbound
at line 460 (NameError). Now caught by the early return from P4.
Co-authored-by: woolcoxm <13604288+woolcoxm@users.noreply.github.com>
FABIOTESS