Feeds reference tokens through the normal step() decode path and reports
NLL/ppl + hit rate + tok/s + RSS. Inert unless PPL=1; default path
untouched (12/12 vs ref.json verified with patch applied). Cross-checked
vs HF transformers bf16 on identical token ids: 12.11 vs 12.25 ppl (#108).
- Fix queue flush race: clear is_queued under mutex only; never move
pilot_w backwards (would break r<=w ring-buffer invariant). Worker
skips stale entries via new is_queued guard at start of pilot_realload
- Add early-exit in pilot_realload when is_queued==0 (entry flushed
between enqueue and worker pickup), preventing unnecessary loads
- Fix misleading EMA struct comment: momentum_logits is used only by
the PILOT prefetcher, not blended into actual MoE routing decisions
- Fix Slot pinned comment: 'never evicted' was too strong; clarify that
pinned slots may be displaced under extreme all-pinned cache pressure
- Use st_read_f32() for scale tensor (.qs) instead of st_read_raw() to
handle potential future BF16/F16 dtype changes robustly
- Fix LRU fallback: when all evictable slots are in-flight, find oldest
non-in-flight slot (pinned ok) before falling back to slot 0
- Fix pin_hot_experts: guard enqueue behind g_pilot>0, call
ensure_pilot_worker_started(), and set is_queued flag to prevent
duplicate in-flight loads from pilot_prefetch()
- Fix token counting: increment token_count/freq_token_count by S (batch
size) instead of 1 so prefill tokens are counted accurately and warmup
threshold triggers at the right time
- Fix ENV VARS header: document PILOT=0-3, SMOOTH, CONF_LIMIT; remove stale REBAL entry
- Fix per-layer EMA: apply routing momentum to all layers (not just layer 0) with correct offset
- Fix in-flight slot race in expert_get: LRU eviction now skips slots with eid==-1 (being loaded)
- Fix in-flight slot race in pilot_realload: same fix, prevents concurrent writes into active slot
- Fix idx[] buffer overflow: clamp max_cand to 128 before E in pilot_prefetch
- load_expert_w: detect I8/U8 tensors and read them raw with their .qs scales.
Before this, pointing SNAP at a convert_olmoe.py container crashed with
SIGBUS (st_read_f32 walks I8 data as 2-byte elements). Container misses now
cost half the I/O and skip quantize_rows entirely: 2.08 -> 3.69 tok/s on the
miss-heavy 16-slot ref.json run (M5). Raw bf16 checkpoints keep working.
- matmul_q: Q8_0-style integer path on ARM (per-16 activation blocks, sdot on
dotprod CPUs, vmull fallback). Same IDOT family as glm.c, same semantics:
IDOT=0 stays byte-exact vs the oracle (12/12); default integer path can flip
an argmax tie (documented in glm.c README). End-to-end on M5: 4.5 -> 12 tok/s
warm-cache decode.
- rss_gb: ru_maxrss is bytes on macOS, KB on Linux. RSS lines were reading
'2029 GB' on Macs.
- convert_olmoe.py: snapshot_download(local_files_only=True) raises
LocalEntryNotFoundError when the repo is not cached; the download fallback
was unreachable.
Measured on M5 MacBook (10 cores, 24 GB, macOS 26.5), OLMoE-1B-7B-0125:
ref.json greedy 12/12 with IDOT=0 on both container and raw paths.
Co-authored-by: x <x@Mac.fritz.box>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
#134: olmoe.c stored experts as int8_t but silently accepted any bits argv;
bits=16 (falsely documented as f32) truncated in quantize_rows -> wraparound
garbage experts. Guard bits to 2..8 with a clear error (int8 is token-exact;
f32 experts are not implemented here, unlike glm.c's fmt=0 path). Doc corrected.
#133: shipped ref.json was from a different checkpoint (continued 'The capital
of the United States is Washington'); the intended target is
allenai/OLMoE-1B-7B-0125-Instruct (per convert_olmoe.py), whose greedy oracle
continues 'The official language of France is French'. full_ids/text updated to
the reporter's verified oracle (engine is already token-exact 12/12 vs it).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
JustVugg