Commit Graph

159 Commits

Author SHA1 Message Date
JustVugg 550ddcba83 glm(win): ignore cmd.exe's built-in PROMPT template; add COLI_PROMPT (#271)
Windows cmd.exe always exports PROMPT (its prompt template, default "$P$G")
into a child's environment. The engine's `getenv("PROMPT")` picked that up, so
running `glm.exe` from cmd for the oracle self-test instead entered text-
generation mode — loading a tokenizer the tiny model doesn't ship and failing
with "tokenizer.json: No such file". PowerShell has no PROMPT env var, so it
worked there (galmok's 32/32) but not in cmd — same command, different shell.

New coli_user_prompt(): honors COLI_PROMPT everywhere, and on Windows ignores a
PROMPT that carries cmd's $-metacodes ($P,$G,...) — a real prompt has none. cmd
users can still pass a prompt via COLI_PROMPT or a non-$ PROMPT. Verified: $P$G
-> oracle mode, "Explain recursion" -> honored, COLI_PROMPT -> honored.

Third native-Windows bug surfaced by a from-scratch main download (after the
-lpsapi link fix and the bilingual oracle message).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-15 23:11:35 +02:00
JustVugg 6b4f5e5fd0 glm: bilingual (EN+IT) oracle-mismatch message, lead with English (#271)
Running a real model with no PROMPT lands in oracle self-test mode, which
compares against ref_glm.json — the TINY model's oracle. The guard that
detects the vocab mismatch and points the user at PROMPT=/coli chat was
Italian-only, which read as a crash to English users (#271, galmok). Lead
with English, keep an IT footer, and add the `coli chat` hint.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-15 23:02:35 +02:00
JustVugg a2391587c7 Makefile(win): link -lpsapi so the MinGW build resolves GetProcessMemoryInfo
compat.h's rss_gb() calls GetProcessMemoryInfo and links psapi via
#pragma comment(lib,"psapi.lib") — an MSVC-ism. MinGW gcc ignores that pragma
(emits -Wunknown-pragmas), and the Windows LDFLAGS never linked psapi, so on a
gcc that doesn't honor the pragma (e.g. 16.1.0 UCRT) the build fails with
`undefined reference to GetProcessMemoryInfo`. Add -lpsapi to the Windows
LDFLAGS; harmless on toolchains where the pragma also resolves it. Found while
building on native Windows 11 with winlibs GCC 16.1.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-15 22:52:17 +02:00
Vincenzo a37dda9eae Merge pull request #289 from woolcoxm/fix/grouped-quant-detect-arbitrary-gs
quant: generalize fmt=4 group-size detection (g64/g128/g256, not just 128)
2026-07-15 22:01:20 +02:00
Vincenzo d9d0e7b958 Merge pull request #288 from monotophic/fix/mux-1token
serve mux: keep ragged decode batches off the fused Metal kernels
2026-07-15 22:00:32 +02:00
woolcoxm 55576941cc quant: generalize fmt=4 group-size detection (g64/g128/g256, not just 128)
The grouped-int4 (fmt=4) loader hard-coded gs=128 at all three detection
sites (resident weights + the two expert-load paths: mmap and pread-slab).
A checkpoint converted with --group-size 64 (or any non-128 size) wrote a
valid file that the engine then misdetected as plain per-row int4 (fmt=2)
and read the wrong number of scales -> garbage output, silently.

The conversion path (convert_fp8_to_int4.py --group-size) already emits
arbitrary group sizes, and the compute kernel (matmul_i4_grouped) is
fully gs-generic (only constraint: gs multiple of 16, the AVX2 width).
The loader was the sole gap.

Replace the three hardcoded checks with one shared helper, detect_group_size(),
that derives gs from the scale-array byte count by probing candidate sizes
{16,32,48,64,96,128,192,256} finest-first. Data-driven: any listed size
just works; per-row int4 (ns == O*4) correctly returns gs=0 -> stays fmt=2.

Verified against real GLM-5.2 expert dims (gate/up O=2048,I=6144 and down
O=6144,I=2048): g64/g128/g256 all detect correctly, and per-row is never
misdetected as grouped. g128 (the only previously-supported size) is
unchanged -> no regression for existing checkpoints.

This unblocks the g64 lane that ZacharyZcR's #225 ablation identified as
beating shipped per-row int4 (-7.5pp vs -9.3pp) at ~19% fewer bits: the
converter can produce it, and now the engine can load it.

Refs #225
2026-07-15 15:54:37 -04:00
Vincenzo 704b4d7eeb Merge pull request #269 from michael-denyer/arm-i8mm-smmla
ARM i8mm: SMMLA fast path for the int8/int4 IDOT drivers (opt-in via ARCH=native)
2026-07-15 16:20:32 +02:00
Michael Denyer d0c9941a67 ARM i8mm: SMMLA fast path for the int8/int4 IDOT drivers
vmmlaq_s32 computes a 2x2 int32 tile (2 weight rows x 2 activation
rows) per instruction on 8-deep segments. Tile o and s in pairs,
halving weight traffic and doubling per-instruction work at S>=2.
Four independent accumulators over a 64-deep unroll keep the loop
throughput-bound (a single chained accumulator measures no better
than SDOT: latency-bound). S=1 and all tails (odd o, odd s, I not a
multiple of 16/32) keep the existing SDOT/scalar code, and scales
apply in the same order, so results are bit-identical.

Compile-time gated on __ARM_FEATURE_MATMUL_INT8. The default Darwin
build passes no -mcpu and is byte-identical (still SDOT, IDOT_KERNEL
"neon"). Opt in with ARCH=native (new Darwin Makefile knob, appends
-mcpu=<arch>), which reports IDOT_KERNEL "neon-i8mm". The same gate
lights up on any aarch64 with i8mm (Graviton3+, Grace).

test_idot grows a driver-level exactness check through matmul_qt_ex:
fmt 1 and 2, S in {2,3,4,5,8}, O in {1,2,3,64,65}, I in {16,17,100,
1408}, bitwise float equality against a plain-C reference. Green on
both build flavors.

Measured on an M5 Pro (18 threads, matmul_qt_ex microbenchmark at
GLM-5.2 expert shapes, best of 3 process runs, vs the SDOT baseline):
  gateup int4 S=8   499.7 -> 1076.6 GF/s  (+115%)
  gateup int8 S=8   512.9 -> 1166.3 GF/s  (+127%)
  down   int4 S=8   696.9 -> 1186.0 GF/s  (+70%)
  S=1 decode rows unchanged (SDOT path untouched)
2026-07-15 14:59:06 +01:00
Vincenzo 52550ff571 Merge pull request #267 from michael-denyer/fix-profile-disk-wait
profile: account expert-disk stall as wait, not other
2026-07-15 15:52:39 +02:00
Michael Denyer da684b240d profile: account expert-disk stall as wait, not other
The decode/prefill PROFILE line splits expert-disk time into service
(overlapped async dispatch) and wait (blocking stalls), but every
accumulation site wrote t_edisk and nothing ever wrote t_ewait, so the
wait column always printed 0.000s. Worse, the accounted sum used the
dead t_ewait instead of t_edisk, so the entire disk-read stall was
excluded from accounted and silently fell into the other bucket.

On a disk-streaming MoE the effect is large: other reads as ~60% of
decode when it is really the expert-load stall. Route the three
blocking sites (non-PIPE parallel load, the Metal drain barrier, and
the per-expert pipe_wait in the CPU matmul loop) to t_ewait, keep the
async dispatch in t_edisk, and include both in accounted.

Profiling-only; no behavioural change. Before, on a 168-expert model:
  expert-disk 25.077s service / 0.000s wait | ... | other 28.401s
After:
  expert-disk 0.111s service / 26.948s wait | ... | other 3.390s
other now holds only the genuinely-unbucketed work (router, norms).
2026-07-15 14:44:29 +01:00
JustVugg 3fdc6d394e Merge remote-tracking branch 'origin/dev' into pr259
# Conflicts:
#	README.md
2026-07-15 15:09:42 +02:00
Vincenzo 0fc18abd24 Merge pull request #265 from skeldoor/feat/interrupt-generation
glm.c/coli: Ctrl-C soft-stops the current turn instead of killing the engine
2026-07-15 14:59:24 +02:00
Vincenzo 87ea13c963 Merge pull request #264 from skeldoor/perf/neon-multiacc-sdot
glm.c: 4-accumulator NEON SDOT for int8/int4 expert dots (2.4x/core on Apple Silicon)
2026-07-15 14:59:12 +02:00
Vincenzo e871fb792d Merge pull request #266 from skeldoor/fix/coli-tiers-leak
coli: consume the engine's TIERS line at startup so it doesn't leak into the first answer
2026-07-15 14:57:08 +02:00
JustVugg da260c38cb server: clamp max_tokens to the server cap instead of rejecting it (fixes #260)
opencode / the ai-sdk OpenAI-compatible client sends a large default max_tokens
(> the server's --max-tokens cap, 1024 by default), and generation_options
returned 400 "must be an integer between 1 and 1024" — even for a trivial
"hello". OpenAI-compatible servers clamp to their own ceiling rather than
reject. Now max_tokens > limit is clamped to limit; only non-int / non-positive
values are a hard error. Test updated to assert the clamp + keep the <1 reject.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-15 14:56:01 +02:00
Skeldoor b79610852b coli: consume the engine's TIERS line at startup so it doesn't leak into the first answer
After the READY handshake, run_serve emits one TIERS status line (the
web-dashboard expert-pyramid snapshot). cmd_chat never read it, so in
interactive chat it surfaced as literal "TIERS 0 181 19275 0.00 ..." text
prepended to the first response. Read and discard that one line after READY,
matching how the other status frames are consumed. Chat-only; the HTTP serve
path already drains it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-15 13:37:36 +01:00
Skeldoor 3745cc680e glm.c/coli: Ctrl-C soft-stops the current turn instead of killing the engine
In serve/chat mode, Ctrl-C during generation killed the whole engine, losing
the loaded model and forcing a full reload. Now a SIGINT handler (armed only
in run_serve / run_serve_mux) sets a flag that spec_decode's token loop treats
exactly like hitting the NGEN cap: the turn ends through the normal path, so
the END sentinel, STAT line, usage_save and KV append all run — and :more can
continue the interrupted answer. The mux loop closes in-flight requests via the
same mux_done path. One-shot ./glm runs and Windows keep default SIGINT (die).

coli: stream_turn survives the first KeyboardInterrupt, forwards SIGINT to the
engine (covers non-TTY), drains to the turn boundary, and reports it. A second
Ctrl-C quits. Help line and per-turn footer updated.

POSIX only (sigaction); no behaviour change on Windows. Verified end-to-end on
Apple M4 + Metal: interrupt mid-decode, engine stays up, next prompt answers,
:q exits 0.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-15 13:37:02 +01:00
Skeldoor af421d6d3a glm.c: 4-accumulator NEON SDOT for int8/int4 expert dots (2.4x/core on Apple Silicon)
dot_i8i8 and dot_i4i8 accumulated the whole SDOT reduction into a single
int32x4_t. SDOT has ~3-4 cycle latency, so the serial dependency on `acc`
capped each core at ~26 GB/s (int8) / ~12 GB/s (int4) of weight throughput
regardless of memory bandwidth. Split into 4 independent accumulators (64
values/iter) so the loads become the bottleneck instead of the reduction
chain; the original single-acc loop is kept as the tail handler.

Measured on an Apple M4 (isolated microbench, expert-shaped 2048x6144):
  int8*int8  26.0 -> 63.2 GB/s/core  (2.4x)
  int4*int8  12.4 -> 29.9 GB/s/core  (2.4x)
Output is bit-identical to the previous kernels (verified over random
inputs). Non-DOTPROD NEON, AVX2/AVX-512/VNNI and VSX paths are untouched;
only the __ARM_FEATURE_DOTPROD branch changed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-15 13:28:05 +01:00
Vincenzo 5d4c3aa11b Merge pull request #257 from woolcoxm/windows-optimizations
Windows disk I/O: pread + PIPE + compat_fadvise (1.70s/tok, mmap reverted)
2026-07-15 13:42:23 +02:00
Vincenzo 4d0438314a Merge pull request #246 from woolcoxm/feat/diskio-kv-batching
Disk I/O: KV cache write batching + persistent file handle
2026-07-15 13:35:03 +02:00
Vincenzo a6895445a9 Merge pull request #254 from woolcoxm/feat/expert-budget
EXPERT_BUDGET=N: miss-aware cap on distinct experts/layer (+75% decode tok/s, 6x prefill on low-RAM hosts)
2026-07-15 13:34:19 +02:00
Vincenzo d9266e39e3 Merge pull request #248 from ZacharyZcR/serve/mux-kv-diag
serve: emit the [API] KV prefix-reuse diagnostic in mux mode (#153)
2026-07-15 13:22:57 +02:00
Vincenzo d99d1a7a0e Merge pull request #255 from ZacharyZcR/tools/quant-e8
tools: E8 lattice quantization in the ablation harness — the missing QuIP# ingredient (#81)
2026-07-15 13:20:44 +02:00
Vincenzo e51ce378ca Merge pull request #245 from ZacharyZcR/tools/atlas-unify
tools: one expert atlas, not two — retire expert_atlas.py, analyze.py gains --web output for the dashboard
2026-07-15 13:20:37 +02:00
Robert Landers 0b9ce242e4 fix wording
Signed-off-by: Robert Landers <landers.robert@gmail.com>
2026-07-15 12:42:13 +02:00
Robert Landers cf062010d8 use async and fix workers
Signed-off-by: Robert Landers <landers.robert@gmail.com>
2026-07-15 12:42:13 +02:00
Robert Landers 5d1eb142ab port uring implementation
Signed-off-by: Robert Landers <landers.robert@gmail.com>
2026-07-15 12:42:10 +02:00
woolcoxm 25219de45b windows: PIPE default ON + compat_fadvise WILLNEED cache-warmer (pread path)
Cut Windows decode disk I/O from 2.06s/tok to 1.70s/tok (budget=4), meeting the
<=2s/tok target. Two changes on the pread expert-load path:

1. compat.h: replace the posix_fadvise no-op with a real WILLNEED cache-warmer
   (overlapped ReadFile into a scratch buffer -> populates the standby page cache
   so the later synchronous pread faults from RAM). Re-arms the existing
   expert_prefetch/PILOT/next-block prefetch chain on Windows. DONTNEED stays a
   no-op (matches macOS; Windows standby-list trimming self-regulates).
   Measured: hit rate 16.4% -> 27.6%.

2. glm.c: flip PIPE (async expert-load thread pool) from default OFF to default ON
   on Windows. Dispatches expert pread onto worker threads so loads overlap the
   matmul, instead of blocking serial load-then-compute. PIPE=0 opts out.
   Measured: expert-disk 65.9s -> 54.3s (-18%).

Also adds compat_fadvise assertions to tests/test_compat_direct.c (data integrity
after cache-warmer, safe no-op on bad fd / non-WILLNEED).

mmap (CreateFileMapping/MapViewOfFile) was implemented and tested at length but
reverted: it regressed on Windows (RSS bloat from touched mapped pages collapsed
the expert cache via ullAvailPhys — a fundamental Windows-vs-Linux difference).
Full findings + the dead-end analysis recorded in issue_diskio.md.
2026-07-15 06:14:15 -04:00
monotophic 0811730845 serve mux: keep ragged decode batches off the fused Metal kernels
With METAL=1 + COLI_METAL=1, run_serve_mux (SERVE_BATCH=1) truncated
every completion to exactly 1 token: step_decode_batch passes per-row
kvs[]/positions[] with pos_base=0, but the two Metal decode fast paths
(attention_rows and the FULL-LAYER CB in layer_forward_rows) ignored
them and dispatched coli_metal_attn_decode/coli_metal_layer_decode with
the model-bound Lc/Rc and the hardcoded pos_base. The kernels' contract
is one sequence, row s at pos_base+s: ragged rows got roped at position
0 and attended over a T=1 window of the wrong cache, so greedy decode
hit a stop token on the first batched step (DONE ... STAT 1).

Gate both fast paths on !kvs. Ragged mux rows now take the CPU absorb
path, which already reads kvs[s]/positions[s]/ks->kv_start per row;
plain serve, chat/run, prefill and MTP verification (kvs==NULL) keep
the fused GPU kernels unchanged.

Verified on GLM-5.2 int4 (M5 Max): mux with COLI_METAL=1 went from
1-token DONE to full 16/16-token greedy completions, byte-identical to
the plain-serve comparator on both test prompts; CPU-only mux was
already correct (bisection); test-c and metal-test pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-15 05:54:50 -04:00
ZacharyZcR 0189a9f0ae tools: E8 lattice quantization for the ablation harness — int{2,3,4,8}[-gN][-e8|-e8u][-rot] (#81) 2026-07-15 17:24:16 +08:00
woolcoxm 9c24b1eceb Merge branch 'feat/expert-budget' into windows-dev 2026-07-15 03:23:25 -04:00
woolcoxm dd0d60692d experiment: miss-aware EXPERT_BUDGET — keep all cache hits, only drop misses
The original budget dropped experts blindly — even cached ones that cost
zero disk I/O. The miss-aware version pre-scans pin/ecache residency
before applying the budget:
  - ALL cache hits are kept (free to compute, no disk I/O)
  - Only misses compete for the remaining budget slots
  - Miss budget = EXPERT_BUDGET - nhits (min 0)

Results (budget=4, same prompt/config as before):
                    Original    Miss-aware
  tok/s             0.33        0.36 (+9%)
  hit rate          16.4%       38.2% (2.3x)
  prefill           8.9s        5.6s (1.6x faster)
  decode            97.4s       88.9s (9% faster)

The hit rate doubling is the key quality signal: the model now gets the
full contribution from all resident experts plus the top-4 new loads,
instead of losing some hits to the budget.
2026-07-15 02:34:58 -04:00
woolcoxm 413b370fbf experiment: EXPERT_BUDGET=N — cap distinct experts/layer, up to 1.8x faster decode
Add EXPERT_BUDGET env var that caps the number of distinct experts loaded
per layer across the batch-union. When the union exceeds the budget, keeps
only the highest-aggregate-gate-weight experts and drops the rest from
idxs[] so they're never loaded from disk.

Complementary to TOPP (per-position) — this trims the cross-position union
that multiplies under MTP/prefill. Based on MoE-Spec (arXiv 2602.16052):
'top 32 of 64 experts capture 93% of routing weight.'

Measurements (GLM-5.2 744B, 24GB RAM, cap=2, MTP=0, 32 tokens):
  Baseline (budget=0):  0.18 tok/s, 9.3% hit,  176s decode, 39s prefill
  EXPERT_BUDGET=12:     0.19 tok/s, 14.0% hit, 171s decode, 12s prefill
  EXPERT_BUDGET=6:      0.26 tok/s, 21.0% hit, 123s decode,  7s prefill
  EXPERT_BUDGET=4:      0.33 tok/s, 16.4% hit,  97s decode,  9s prefill

Budget=4 nearly doubles decode speed (+83%) and 4x's prefill speed by
halving disk reads per layer. Default OFF (EXPERT_BUDGET=0).
2026-07-15 02:34:57 -04:00
woolcoxm 71c262ce1a diskio: KV write batching + persistent handle; generators: unfuse experts
Two independent fixes validated end-to-end on fresh fixtures:

1. KV cache disk I/O (issue_diskio.md opportunities #1 + #4):
   - kv_disk_append: fopen/fclose every turn -> persistent FILE* kept open
     for the engine lifetime, lazy open on first append, closed in
     serve_ctx_free. Eliminates per-turn handle creation overhead.
   - kv_disk_append: ~157 small fwrites per position -> one contiguous
     record memcpy'd into a staging buffer then a single fwrite per
     position. The staging buffer grows on demand via realloc.
   - kv_disk_truncate: closes the persistent handle before truncating
     so the file actually shrinks on disc, then reopens lazily.
   - KVState gains disk_fp, disk_buf, disk_buf_cap fields.
   - Verified: serve-mode round-trip, write 11 tokens then reload and
     resume with no re-prefill, then append 8 more and reload to 19.

2. Expert weight unfusing in test-model generators:
   - The real GLM-5.2-FP8 checkpoint stores routed experts UNFUSED as
     per-expert 2-D tensors, each with its own _scale_inv. HF fuses
     gate+up into a single 3-D gate_up_proj for compute efficiency.
   - The converter and C engine both expect the unfused layout. The
     fused 3-D tensors were silently skipped by the converter, and the
     engine crashed with missing-tensor errors.
   - New unfuse_experts in glm_fp8_emit.py splits gate_up_proj and
     down_proj into per-expert 2-D tensors. Called after reference
     generation but before saving, in both generators, both FP8 and bf16.
   - Also fixed: make_glm_oracle.py FP8 round-trip guard used p.dim()<2
     which let 3-D fused experts through and crashed fp8_block_quantize.
     Changed to p.dim()!=2 to match the converter ndim!=2 guard.

Validated full chain on fresh fixtures:
  generator --fp8 -> 570 e4m3 tensors + 629 scale_inv, was 90 when fused
  converter --group-size 0  -> per-row int4 fmt=2, engine loads clean
  converter --group-size 128 -> grouped int4 fmt=4, 8-16x more scales,
    engine loads clean, fmt=4 auto-detected in both mmap and slab paths
  dequant error: grouped 1.14-1.22x lower than per-row vs FP8 source
2026-07-15 02:32:12 -04:00
woolcoxm e71d4fbe29 windows-dev: CUDA build script, expert budget + grouped quant research
- build_cuda.bat: one-shot nvcc DLL compilation for sm_120 (RTX 5070 Ti)
- issue_budget.md: EXPERT_BUDGET research (cap distinct experts/layer, 2x tok/s)
- issue_grouped_quant.md: root cause of int4 incoherence (per-row vs group-128 scales)
- glm_fp8_emit.py: FP8 e4m3 test weight generator for converter validation
2026-07-15 02:32:12 -04:00
woolcoxm 2c6946c478 test-models: add --fp8 emission for the FP8->int4 converter test path
Both test-model generators (make_glm_oracle.py, make_glm_bench_model.py) can now
emit weights as FP8 e4m3 + 128x128 block scale_inv, in the same layout as the real
GLM-5.2-FP8 checkpoint. This lets convert_fp8_to_int4.py exercise its FP8->int4
dequant path on a local fixture without the 379 GB download.

- New shared helper glm_fp8_emit.py: FP8 block quantize/dequantize (FBGEMM/TE
  scale=amax/448 convention) + state_dict emitter. Only exactly-2-D tensors are
  quantized; 1-D/3-D and norms/router/e_score_correction_bias are kept as f32,
  mirroring the converter's classify() + ndim!=2 guard.
- make_glm_bench_model.py: opt-in --fp8 writes model.safetensors in FP8 layout
  (config.json written explicitly since the FP8 path bypasses save_pretrained);
  manifest gains a 'format' field. Default bf16 behavior unchanged.
- make_glm_oracle.py: opt-in --fp8 round-trips quantizable weights through FP8
  before computing ref_glm.json, so the reference reflects exactly the FP8 model
  the converter ingests. Default bf16 oracle contract unchanged.

Verified end-to-end: FP8 model -> converter --indir -> int4 U8 + .qs F32 output,
bit-identical dequant between helper and converter (maxdiff 0.0).
2026-07-15 02:32:12 -04:00
woolcoxm 69f65d5173 Windows-dev: grouped quantization + mixed precision + expert budget + download tool
Consolidates all experiment branches into one Windows-dev branch:

1. Group-scaled int4 (fmt=4, gs=128) — glm.c
   - QT struct: added gs field
   - matmul_i4_grouped: AVX2 kernel, verified to 3e-08 vs f32
   - Format detection: auto-detects from .qs scale array size
   - expert_load: both mmap and slab+pread paths handle fmt=4
   - qt_bytes: fmt=4 case added

2. Per-tensor-type mixed precision — convert_fp8_to_int4.py
   - Split classify() into sh/o/kvb/attn/dmlp sub-types
   - New args: --shared-bits, --o-bits, --kvb-bits, --attn-bits, --dmlp-bits
   - Plan: shared expert + o_proj + kv_b_proj at int8, rest grouped int4
   - Only +5.3 GB RAM vs +0 for pure int4

3. EXPERT_BUDGET (miss-aware) — glm.c
   - Caps distinct experts per layer across batch-union
   - Always keeps cache hits, only drops misses
   - Up to 1.8x faster decode on low-RAM hosts

4. Two-step shared-expert prediction (PILOT_TWO) — glm.c
   - la_predict kind==2 + pilot_prefetch integration
   - +3.1% recall over baseline PILOT

5. FP8 download tool — download_fp8.py
   - ModelScope + HuggingFace dual-source
   - Parallel shard download with stall recovery

6. Tiny model generation — make_glm_oracle.py
   - Generated and tested locally for pipeline validation
2026-07-15 02:32:12 -04:00
woolcoxm e141db047d converter: per-tensor-type mixed-precision control
Split the resident weight classification into 5 sub-types so each can
get different precision:
  sh   = shared expert (highest sensitivity, fires every token)
  o    = o_proj (reconstructs output, biggest attn tensor)
  kvb  = kv_b_proj (reconstructs KV cache on every decode)
  attn = q_a/q_b/kv_a (other attention projections)
  dmlp = dense MLP (first 3 layers)

New args: --shared-bits, --o-bits, --kvb-bits, --attn-bits, --dmlp-bits
Each defaults to ebits (backward compat). When set, the converter applies
that precision to just that tensor type.

Research-backed plan: put the 3 compounding tensors (shared expert, o_proj,
kv_b_proj) at int8 and everything else at grouped int4. Extra RAM cost:
only +5.3 GB (those tensors are small vs the 372 GB expert pool on disk).
2026-07-15 02:32:12 -04:00
ZacharyZcR d04d99e039 serve: print the KV prefix-reuse diagnostic in mux mode too — [API] KV slot line existed only in the legacy \x02PROMPT path (#153) 2026-07-15 14:31:36 +08:00
woolcoxm e2ad6c72e6 diskio: KV write batching + persistent handle; generators: unfuse experts
Two independent fixes validated end-to-end on fresh fixtures:

1. KV cache disk I/O (issue_diskio.md opportunities #1 + #4):
   - kv_disk_append: fopen/fclose every turn -> persistent FILE* kept open
     for the engine lifetime, lazy open on first append, closed in
     serve_ctx_free. Eliminates per-turn handle creation overhead.
   - kv_disk_append: ~157 small fwrites per position -> one contiguous
     record memcpy'd into a staging buffer then a single fwrite per
     position. The staging buffer grows on demand via realloc.
   - kv_disk_truncate: closes the persistent handle before truncating
     so the file actually shrinks on disc, then reopens lazily.
   - KVState gains disk_fp, disk_buf, disk_buf_cap fields.
   - Verified: serve-mode round-trip, write 11 tokens then reload and
     resume with no re-prefill, then append 8 more and reload to 19.

2. Expert weight unfusing in test-model generators:
   - The real GLM-5.2-FP8 checkpoint stores routed experts UNFUSED as
     per-expert 2-D tensors, each with its own _scale_inv. HF fuses
     gate+up into a single 3-D gate_up_proj for compute efficiency.
   - The converter and C engine both expect the unfused layout. The
     fused 3-D tensors were silently skipped by the converter, and the
     engine crashed with missing-tensor errors.
   - New unfuse_experts in glm_fp8_emit.py splits gate_up_proj and
     down_proj into per-expert 2-D tensors. Called after reference
     generation but before saving, in both generators, both FP8 and bf16.
   - Also fixed: make_glm_oracle.py FP8 round-trip guard used p.dim()<2
     which let 3-D fused experts through and crashed fp8_block_quantize.
     Changed to p.dim()!=2 to match the converter ndim!=2 guard.

Validated full chain on fresh fixtures:
  generator --fp8 -> 570 e4m3 tensors + 629 scale_inv, was 90 when fused
  converter --group-size 0  -> per-row int4 fmt=2, engine loads clean
  converter --group-size 128 -> grouped int4 fmt=4, 8-16x more scales,
    engine loads clean, fmt=4 auto-detected in both mmap and slab paths
  dequant error: grouped 1.14-1.22x lower than per-row vs FP8 source
2026-07-15 02:28:33 -04:00
Vincenzo 3fd47b7bbd Merge pull request #242 from woolcoxm/feat/grouped-quant-fmt4
Group-scaled int4 (fmt=4): one scale per 128 elements — fixes incoherent output
2026-07-15 08:25:29 +02:00
ZacharyZcR e7cb7df501 tools: one expert atlas, not two — retire expert_atlas.py, analyze.py gains --web for the dashboard 2026-07-15 14:16:36 +08:00
JustVugg ff04b320d0 security(win): load coli_cuda.dll by absolute path, never from the CWD (DLL hijack)
coli_cuda_load did LoadLibraryA("coli_cuda.dll") with a bare name. Windows'
default search order includes the current working directory (and, without
SafeDllSearchMode, other writable locations), so an attacker who plants a
coli_cuda.dll where the user launches glm.exe — or inside a downloaded model
directory the user cd's into — gets their DllMain executed at load time:
DLL hijacking -> arbitrary code execution.

Now the loader resolves the path next to glm.exe via GetModuleFileNameA and
loads that exact file with LOAD_WITH_ALTERED_SEARCH_PATH, so both the DLL and
its dependency search are anchored to the trusted install directory. Fallback
(if GetModuleFileNameA ever fails) uses LOAD_LIBRARY_SEARCH_APPLICATION_DIR |
LOAD_LIBRARY_SEARCH_SYSTEM32 — which also excludes the CWD. Cross-compiles clean
under mingw-w64; the CPU path is unaffected (this file is _WIN32-only).

Note: grammar.h gr__rule memcpy was reviewed in the same pass and is safe (len
is clamped to 63 into a name[64] buffer).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-15 08:16:20 +02:00
JustVugg 4268e00fa1 security: harden safetensors/JSON parsers against malicious model files
A downloaded (supply-chain) model file was fully trusted by the loader. Three
memory-safety holes, all reachable by pointing the engine at a crafted shard —
demonstrated crashing on pre-fix, now rejected fail-closed:

st.h (safetensors):
- header length `hlen` (u64 from the file) was unbounded before malloc(hlen+1):
  a crafted value overflows (malloc(0) then hdr[hlen]=0 OOB) or forces a giant
  allocation. Now bounded to the file size and a 512 MB cap; malloc NULL-checked.
- json_get() returns NULL for missing/mistyped fields, but dtype/data_offsets/
  shape were dereferenced blind (off->kids[0]) — a header omitting data_offsets
  SIGSEGV'd (verified). Now type/arity-checked before use.
- data_offsets [a0,b0] were trusted: b0<a0 gave a negative nbytes -> malloc((size_t))
  giant and an oversized memcpy into the caller's buffer in st_read_f32 (heap
  overflow); off could point outside the file. Now validated 0<=a0<=b0 and
  data_start+b0<=filesize.

json.h: j_parse_val recursed with no depth limit -> stack overflow on nested
input like [[[[...]]]]. Added J_MAX_DEPTH=1024 (headers are ~3 deep); wide-but-
flat objects like the GLM header are unaffected (depth is decremented per return).

eval_glm.py: tempfile.mktemp() -> mkstemp() — closes the TOCTOU/symlink race on
a shared tmp dir (CWE-377).

Network path (openai_server.py + serve SUBMIT parser) audited separately and is
already sound: hmac.compare_digest auth, MAX_BODY cap, resolve()+relative_to
traversal guard, list-form subprocess, bounded/validated SUBMIT header. All 62
tests pass; valid GLM/OLMoE shards load unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-15 08:13:20 +02:00
woolcoxm 498ab0c20e experiment: group-scaled int4 (fmt=4) — one scale per 128 elements, not per row
Root cause of gibberish output: the int4 quantization uses one F32 scale per
output row (2048 scales for a 2048x6144 matrix). The FP8 source has 128x128
block scales — 48x finer. This destroys reasoning while keeping surface fluency.

Changes:
- Converter: quant_int4_grouped() with --group-size 128 arg. Same nibble
  packing, but one scale per group of 128 elements along the input dim.
- Engine QT struct: added 'gs' field (group size, 0=per-row backward compat)
- Engine qt_from_disk: auto-detects fmt=4 when scale array is O*ceil(I/128)
  elements instead of O. Old per-row models (fmt=2) work unchanged.
- Engine matmul_i4_grouped(): AVX2 kernel that applies per-group scales.
  Accumulator resets at each group boundary: dot(x[grp],w[grp]) * scale[grp].
- Engine matmul_qt_ex: dispatches to grouped kernel for fmt=4 (always exact,
  no IDOT approximation since the point is quality)
- Engine expert_load: both mmap and slab+pread paths detect fmt=4 from
  scale array size and set gs=128
- qt_bytes: fmt=4 reports correct memory including group scales

Backward compatible: existing per-row int4 models work unchanged.
The fused gate+up pair path (matmul_i4_pair) falls back to separate
matmul_qt calls for fmt=4 — minor perf cost, correctness preserved.
2026-07-15 02:05:18 -04:00
Vincenzo a8895f2d84 Merge pull request #193 from bokiko/feature/disk-split
stats: opt-in disk-load split by speculation context and layer kind (DISK_SPLIT=1)
2026-07-15 07:59:33 +02:00
JustVugg c71e5d1cf8 glm: honest short-read reporting in expert pread; Makefile: -pthread for *BSD
Two small correctness fixes surfaced by community reports on non-Linux hosts:

#236 — expert_load's buffered pread paths (slab + qs scales) used
perror("pread expert") on a short read. Since pread returned a short count
(not -1), errno stays 0 and perror prints "Success" — a confusing message
right before exit(1) in the score/bench path. New pread_full() helper loops
over short reads and EINTR and reports actual/expected bytes and offset, so a
truncated shard reads as such instead of "Success".

#219 — Linux pulls -pthread in via -fopenmp; the *BSDs do not, so pthread_*
fail to link there. Added -pthread to the generic (Linux/*BSD x86-64) and
aarch64 CFLAGS/LDFLAGS. No-op on Linux, required on FreeBSD (complements #206).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-15 07:55:09 +02:00
JustVugg ddf9212b81 glm: PIN_GB=all clamps to the --ram budget instead of pinning every expert (fixes #229)
PIN_GB=all passed gb=-1.0 to pin_load, which took npin=n (all experts) and
ignored --ram entirely — the OOM-kill regression from #80. A 92 GB host with
--ram 78 was killed mid-generation (anon-rss ~89 GB). Now gb<0 clamps npin to
expert_avail() — how many experts fit the RAM budget, same accounting AUTOPIN
uses. pin_load already adds the pinned bytes to resident_bytes, so the later
cap_for_ram narrows the LRU accordingly with no double count.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-15 07:50:48 +02:00
Vincenzo 22c91403b7 Merge pull request #228 from bokiko/feature/score-prefix-default
engine: SCORE mode defaults the [gMASK]<sop> prefix ON for GLM snapshots (#194 follow-up, #108)
2026-07-15 07:47:46 +02:00
Vincenzo 00902a2ea9 Merge pull request #224 from michael-denyer/fix-kv-alloc-double-free
glm: fix kv_alloc double-free on KV re-allocation (stale pre-Metal free block)
2026-07-15 07:47:16 +02:00