Feeds reference tokens through the normal step() decode path and reports
NLL/ppl + hit rate + tok/s + RSS. Inert unless PPL=1; default path
untouched (12/12 vs ref.json verified with patch applied). Cross-checked
vs HF transformers bf16 on identical token ids: 12.11 vs 12.25 ppl (#108).
- load_expert_w: detect I8/U8 tensors and read them raw with their .qs scales.
Before this, pointing SNAP at a convert_olmoe.py container crashed with
SIGBUS (st_read_f32 walks I8 data as 2-byte elements). Container misses now
cost half the I/O and skip quantize_rows entirely: 2.08 -> 3.69 tok/s on the
miss-heavy 16-slot ref.json run (M5). Raw bf16 checkpoints keep working.
- matmul_q: Q8_0-style integer path on ARM (per-16 activation blocks, sdot on
dotprod CPUs, vmull fallback). Same IDOT family as glm.c, same semantics:
IDOT=0 stays byte-exact vs the oracle (12/12); default integer path can flip
an argmax tie (documented in glm.c README). End-to-end on M5: 4.5 -> 12 tok/s
warm-cache decode.
- rss_gb: ru_maxrss is bytes on macOS, KB on Linux. RSS lines were reading
'2029 GB' on Macs.
- convert_olmoe.py: snapshot_download(local_files_only=True) raises
LocalEntryNotFoundError when the repo is not cached; the download fallback
was unreachable.
Measured on M5 MacBook (10 cores, 24 GB, macOS 26.5), OLMoE-1B-7B-0125:
ref.json greedy 12/12 with IDOT=0 on both container and raw paths.
Co-authored-by: x <x@Mac.fritz.box>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
#134: olmoe.c stored experts as int8_t but silently accepted any bits argv;
bits=16 (falsely documented as f32) truncated in quantize_rows -> wraparound
garbage experts. Guard bits to 2..8 with a clear error (int8 is token-exact;
f32 experts are not implemented here, unlike glm.c's fmt=0 path). Doc corrected.
#133: shipped ref.json was from a different checkpoint (continued 'The capital
of the United States is Washington'); the intended target is
allenai/OLMoE-1B-7B-0125-Instruct (per convert_olmoe.py), whose greedy oracle
continues 'The official language of France is French'. full_ids/text updated to
the reporter's verified oracle (engine is already token-exact 12/12 vs it).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Peter Groom