olmoe: int8 container support (fixes SIGBUS), NEON int8 dot kernels (2.7x on ARM), macOS RSS fix, converter fallback (#187)
- load_expert_w: detect I8/U8 tensors and read them raw with their .qs scales. Before this, pointing SNAP at a convert_olmoe.py container crashed with SIGBUS (st_read_f32 walks I8 data as 2-byte elements). Container misses now cost half the I/O and skip quantize_rows entirely: 2.08 -> 3.69 tok/s on the miss-heavy 16-slot ref.json run (M5). Raw bf16 checkpoints keep working. - matmul_q: Q8_0-style integer path on ARM (per-16 activation blocks, sdot on dotprod CPUs, vmull fallback). Same IDOT family as glm.c, same semantics: IDOT=0 stays byte-exact vs the oracle (12/12); default integer path can flip an argmax tie (documented in glm.c README). End-to-end on M5: 4.5 -> 12 tok/s warm-cache decode. - rss_gb: ru_maxrss is bytes on macOS, KB on Linux. RSS lines were reading '2029 GB' on Macs. - convert_olmoe.py: snapshot_download(local_files_only=True) raises LocalEntryNotFoundError when the repo is not cached; the download fallback was unreachable. Measured on M5 MacBook (10 cores, 24 GB, macOS 26.5), OLMoE-1B-7B-0125: ref.json greedy 12/12 with IDOT=0 on both container and raw paths. Co-authored-by: x <x@Mac.fritz.box> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -55,9 +55,13 @@ def main():
|
||||
|
||||
if args.repo:
|
||||
from huggingface_hub import snapshot_download
|
||||
from huggingface_hub.errors import LocalEntryNotFoundError
|
||||
print(f"Downloading {args.repo}...")
|
||||
src_dir = snapshot_download(args.repo, local_files_only=True, max_workers=4)
|
||||
if not any(Path(src_dir).glob("*.safetensors")):
|
||||
try:
|
||||
src_dir = snapshot_download(args.repo, local_files_only=True, max_workers=4)
|
||||
except LocalEntryNotFoundError:
|
||||
src_dir = None
|
||||
if src_dir is None or not any(Path(src_dir).glob("*.safetensors")):
|
||||
print("Downloading safetensors...")
|
||||
src_dir = snapshot_download(args.repo, max_workers=4)
|
||||
else:
|
||||
|
||||
Reference in New Issue
Block a user