Three crash bugs in convert_fp8_to_int4.py, all on the download path:
P3: 'import fcntl' at line 211 is Unix-only — ModuleNotFoundError on Windows.
The --indir test path returns before reaching it so tests pass, but
'--repo' on Windows hard-crashes. Guarded the import: try fcntl (Unix),
fall back to msvcrt.locking (Windows), skip if neither available.
P4: repo_info retry loop had range(999) — up to ~16 hours of retries on a
bad network, then fell through to line 395 where 'info' was unbound
(NameError). Capped at 10 retries and added an explicit error + return
when exhausted. Also added an early return if no safetensors shards are
found in the repo.
P5: if the shards list was empty (wrong repo, all filtered out), the
'for i, sh in enumerate(shards)' loop never executed and 'i' was unbound
at line 460 (NameError). Now caught by the early return from P4.
Co-authored-by: woolcoxm <13604288+woolcoxm@users.noreply.github.com>
The tool tokenized every context with add_special_tokens=False, so pointing it at a
GLM snapshot scored the model out-of-distribution — the same bug @bokiko found in
eval_glm.py (#108).
Measured on this box (GLM-5.2 int4, engine SCORE mode, same 1023 target tokens,
only the conditioning prefix changed):
corpus no prefix +[gMASK]<sop>
natural prose PPL 29.2 PPL 9.4
markdown + code PPL 131.0 PPL 24.5
It does not just depress scores, it distorts sensitivity to numerical changes: an
exact-vs-IDOT kernel A/B measured without the prefix reported a penalty that halved
AND flipped sign on one of two corpora once the prefix was restored (#153). A
quantization-ablation tool that scores OOD is therefore worse than useless — it
produces confident deltas that are artifacts.
The GLM tokenizer does not add the prefix itself (add_special_tokens=True is a no-op
there), so it must be prepended explicitly. Auto-detected from the vocab rather than
opt-in, so it cannot be lost by omission; --prefix overrides. Models with no such
prefix are unaffected: OLMoE (the tool's default) has no BOS at all, add_special_tokens
True/False give identical ids, so the numbers in #108 measured on OLMoE still stand.
Problem: 'make clean', 'make test-c', and 'make check' use POSIX shell
constructs (for loop, rm -f, rm -rf) that require sh.exe. On native Windows
with WinLibs MinGW (no MSYS2, no Git Bash), there is no sh.exe on PATH.
GNU Make falls back to cmd.exe, which can't parse 'for test in ...; do'
or find 'rm', so these targets fail with 'test was unexpected' or
'CreateProcess error'.
Root cause: the Makefile's recipe lines assumed a POSIX shell is always
available. The IS_WIN detection (from #129) catches the platform but the
shell-dependent targets were never made portable.
Fix: replace the shell-dependent constructs with small Python helper scripts
(Python is already a project dependency for test-python, convert, bench).
This works from cmd.exe, PowerShell, Git Bash, and MSYS2 alike.
Changes:
- tools/run_tests.py (new): runs each C test binary, exits non-zero on the
first failure. Replaces the 'for test in ...; do ./$test || exit 1; done'
shell loop in test-c.
- tools/clean.py (new): removes build artifacts and test binaries. Replaces
'rm -f' and 'rm -rf' in clean. Only removes executables (.exe) and known
artifact names — never .c or .py source files.
- Makefile: PYTHON defaults to 'python' on Windows (not 'python3'); test-c
and clean now call the Python helpers instead of shell constructs.
Verified from native PowerShell (no sh.exe): make clean removes 8-19
files/dirs, make test-c runs all 7 C test suites, source files survive.
Also verified from Git Bash (sh.exe present): behavior unchanged.
Co-authored-by: woolcoxm <13604288+woolcoxm@users.noreply.github.com>
Extends the ablation grammar to int{2,3,4,8}[-g<N>][-rot][-nohead]. -rot round-trips weights through an orthogonal Hadamard Q=diag(±1)·H/√n on the input dim, measuring the exact weight error of a deployed rotate-activations scheme. Engine-free tool (c/tools/quant_ablation.py only). Verified: syntax clean, scheme parser correct (int3/-rot/-nohead), no unsafe constructs. Findings feed the v2 int3-g64 direction (#81/#108).
Measuring "what does int4 cost?" by comparing colibri's score to a published
model-card number does not work: this harness scores 0-shot log-likelihood while
published numbers are few-shot/CoT, and that protocol gap can swamp the
quantization effect entirely (#108).
This removes the confound by construction: take an fp16 model, push its weights
through colibri's own quantizer (quantize -> dequantize, in place), and score both
with the SAME harness on the SAME questions. The only variable is the quantizer, so
the delta IS the quantization cost. Runs on OLMoE in minutes, so a scheme can be
ranked BEFORE committing to a multi-hour GLM conversion.
Quantizer math is replicated from tools/convert_fp8_to_int4.py (symmetric absmax,
per-row scales) and generalised with an optional group size, so grouped/finer schemes
can be compared directly against what ships today.
Measured on OLMoE-1B-7B, n=200/task (#108):
scheme hellaswag arc_c mmlu mean delta
fp16 77.0% 47.0% 47.0% 57.0% --
int4 (shipped) 74.0% 41.0% 31.5% 48.8% -8.2pp
int4-nohead 73.5% 40.5% 37.5% 50.5% -6.5pp
int4-g128 78.5% 45.5% 38.0% 54.0% -3.0pp
int4-g128-nohead 78.5% 46.5% 38.0% 54.3% -2.7pp
The per-row int4 container costs ~8pp, concentrated on the HARD task: MMLU falls to
31.5% against a 25% random baseline while easy HellaSwag barely moves -- per-row
scales eat the small logit margins that hard questions depend on (the same margin
erosion that flips near-tie tokens in #100). group=128 recovers ~63% of the loss.
Keeping lm_head/embed in fp16 is NOT the fix (+1.7pp alone, +0.3pp atop grouping).
Includes a coverage assert: transformers fuses MoE experts into 3D tensors, so a
ndim==2 filter silently skips every expert and leaves ~85% of the model in fp16 while
appearing to work. The tool fails loudly instead of reporting fiction.
Dev-only tool (torch + transformers); the engine's dependency-free path is untouched.
woolcoxm