olmoe: int8 container support (fixes SIGBUS), NEON int8 dot kernels (2.7x on ARM), macOS RSS fix, converter fallback (#187)

- load_expert_w: detect I8/U8 tensors and read them raw with their .qs scales.
  Before this, pointing SNAP at a convert_olmoe.py container crashed with
  SIGBUS (st_read_f32 walks I8 data as 2-byte elements). Container misses now
  cost half the I/O and skip quantize_rows entirely: 2.08 -> 3.69 tok/s on the
  miss-heavy 16-slot ref.json run (M5). Raw bf16 checkpoints keep working.
- matmul_q: Q8_0-style integer path on ARM (per-16 activation blocks, sdot on
  dotprod CPUs, vmull fallback). Same IDOT family as glm.c, same semantics:
  IDOT=0 stays byte-exact vs the oracle (12/12); default integer path can flip
  an argmax tie (documented in glm.c README). End-to-end on M5: 4.5 -> 12 tok/s
  warm-cache decode.
- rss_gb: ru_maxrss is bytes on macOS, KB on Linux. RSS lines were reading
  '2029 GB' on Macs.
- convert_olmoe.py: snapshot_download(local_files_only=True) raises
  LocalEntryNotFoundError when the repo is not cached; the download fallback
  was unreachable.

Measured on M5 MacBook (10 cores, 24 GB, macOS 26.5), OLMoE-1B-7B-0125:
ref.json greedy 12/12 with IDOT=0 on both container and raw paths.

Co-authored-by: x <x@Mac.fritz.box>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Louie
2026-07-14 23:59:49 +10:00
committed by GitHub
parent 32629eae7f
commit ec0aadf91a
2 changed files with 62 additions and 5 deletions
+6 -2
View File
@@ -55,9 +55,13 @@ def main():
if args.repo:
from huggingface_hub import snapshot_download
from huggingface_hub.errors import LocalEntryNotFoundError
print(f"Downloading {args.repo}...")
src_dir = snapshot_download(args.repo, local_files_only=True, max_workers=4)
if not any(Path(src_dir).glob("*.safetensors")):
try:
src_dir = snapshot_download(args.repo, local_files_only=True, max_workers=4)
except LocalEntryNotFoundError:
src_dir = None
if src_dir is None or not any(Path(src_dir).glob("*.safetensors")):
print("Downloading safetensors...")
src_dir = snapshot_download(args.repo, max_workers=4)
else: