woolcoxm 69f65d5173 Windows-dev: grouped quantization + mixed precision + expert budget + download tool ...

Consolidates all experiment branches into one Windows-dev branch:

1. Group-scaled int4 (fmt=4, gs=128) — glm.c
   - QT struct: added gs field
   - matmul_i4_grouped: AVX2 kernel, verified to 3e-08 vs f32
   - Format detection: auto-detects from .qs scale array size
   - expert_load: both mmap and slab+pread paths handle fmt=4
   - qt_bytes: fmt=4 case added

2. Per-tensor-type mixed precision — convert_fp8_to_int4.py
   - Split classify() into sh/o/kvb/attn/dmlp sub-types
   - New args: --shared-bits, --o-bits, --kvb-bits, --attn-bits, --dmlp-bits
   - Plan: shared expert + o_proj + kv_b_proj at int8, rest grouped int4
   - Only +5.3 GB RAM vs +0 for pure int4

3. EXPERT_BUDGET (miss-aware) — glm.c
   - Caps distinct experts per layer across batch-union
   - Always keeps cache hits, only drops misses
   - Up to 1.8x faster decode on low-RAM hosts

4. Two-step shared-expert prediction (PILOT_TWO) — glm.c
   - la_predict kind==2 + pilot_prefetch integration
   - +3.1% recall over baseline PILOT

5. FP8 download tool — download_fp8.py
   - ModelScope + HuggingFace dual-source
   - Parallel shard download with stall recovery

6. Tiny model generation — make_glm_oracle.py
   - Generated and tested locally for pipeline validation

2026-07-15 02:32:12 -04:00

..

scripts

Translate user-facing runtime output to English, machine prefixes preserved, + CLI output test (#67, #85)

2026-07-12 13:38:40 +02:00

tests

Merge pull request #224 from michael-denyer/fix-kv-alloc-double-free

2026-07-15 07:47:16 +02:00

tools

Windows-dev: grouped quantization + mixed precision + expert budget + download tool

2026-07-15 02:32:12 -04:00

.gitignore

Windows 11 native port, phase 1: MinGW-w64 static build, compat shims, setup + docs (#40)

2026-07-11 12:59:49 +02:00

backend_cuda.cu

GPU resident pipeline: batch CUDA attention, head-sharded kv_b, prefill expert groups, W4A16 mixed dispatch (#111)

2026-07-14 18:18:05 +02:00

backend_cuda.h

cuda(win): export + load the #111 pipeline ABI — 24 missing DLLEXPORT decorations + loader wrappers, unbreaks CUDA_DLL=1 (#201)

2026-07-14 18:43:56 +02:00

backend_loader.c

security(win): load coli_cuda.dll by absolute path, never from the CWD (DLL hijack)

2026-07-15 08:16:20 +02:00

backend_metal.h

Metal backend (Apple Silicon): batched experts + fused attention on GPU, unified-memory zero-copy, gated behind COLI_METAL — 2.06 tok/s M5 Max (#72, #87, #103)

2026-07-13 08:57:10 +02:00

backend_metal.mm

Metal backend (Apple Silicon): batched experts + fused attention on GPU, unified-memory zero-copy, gated behind COLI_METAL — 2.06 tok/s M5 Max (#72, #87, #103)

2026-07-13 08:57:10 +02:00

coli

bench: pass the resolved GLM engine path to eval_glm.py — cwd-relative ./glm default broke coli bench from other dirs (#203)

2026-07-14 18:55:36 +02:00

compat.h

Windows: direct I/O (FILE_FLAG_NO_BUFFERING, 1.47x decode) + VirtualLock pin wiring + select_ctx device cache (#162)

2026-07-14 16:12:15 +02:00

decode_batch.h

Unify continuous batching + heterogeneous runtime: decode batching, physical-core planning, disjoint VRAM/RAM placement, topp-policy warning (CPU-validated, CUDA on 6x5090) (#68)

2026-07-13 14:30:36 +02:00

doctor.py

config: fix UTF-8 decode crash + null config-value TypeError on Windows (doctor/resource_plan) (#184)

2026-07-14 14:17:42 +02:00

download_fp8.py

Windows-dev: grouped quantization + mixed precision + expert budget + download tool

2026-07-15 02:32:12 -04:00

glm.c

Merge pull request #242 from woolcoxm/feat/grouped-quant-fmt4

2026-07-15 08:25:29 +02:00

grammar.h

Translate user-facing runtime output to English, machine prefixes preserved, + CLI output test (#67, #85)

2026-07-12 13:38:40 +02:00

iobench.c

Windows: direct I/O (FILE_FLAG_NO_BUFFERING, 1.47x decode) + VirtualLock pin wiring + select_ctx device cache (#162)

2026-07-14 16:12:15 +02:00

json.h

security: harden safetensors/JSON parsers against malicious model files

2026-07-15 08:13:20 +02:00

Makefile

glm: honest short-read reporting in expert pread; Makefile: -pthread for *BSD

2026-07-15 07:55:09 +02:00

olmoe.c

FreeBSD compatibility: platform guards in glm.c/olmoe.c (#206)

2026-07-14 22:08:12 +02:00

openai_server.py

openai_server: fix path traversal in web static serving (relative_to, not startswith) + empty-prompt validation, with tests (#212)

2026-07-14 22:07:41 +02:00

ref_glm.json

colibrì: pure-C GLM-5.2 (744B MoE) engine with disk-streamed experts

2026-07-05 20:57:25 +02:00

resource_plan.py

config: fix UTF-8 decode crash + null config-value TypeError on Windows (doctor/resource_plan) (#184)

2026-07-14 14:17:42 +02:00

schema_gbnf.h

SCHEMA=<file.json>: JSON-Schema -> GBNF compiler for grammar-forced drafts (#148)

2026-07-14 16:31:09 +02:00

setup.sh

Translate user-facing runtime output to English, machine prefixes preserved, + CLI output test (#67, #85)

2026-07-12 13:38:40 +02:00

st.h

security: harden safetensors/JSON parsers against malicious model files

2026-07-15 08:13:20 +02:00

tier.h

Unify continuous batching + heterogeneous runtime: decode batching, physical-core planning, disjoint VRAM/RAM placement, topp-policy warning (CPU-validated, CUDA on 6x5090) (#68)

2026-07-13 14:30:36 +02:00

tok_unicode.h

Translate user-facing runtime output to English, machine prefixes preserved, + CLI output test (#67, #85)

2026-07-12 13:38:40 +02:00

tok.h

Translate user-facing runtime output to English, machine prefixes preserved, + CLI output test (#67, #85)

2026-07-12 13:38:40 +02:00

warmup.ps1

Windows native port: serve-mode pipe fix + RAM detection + POSIX guards, AVX-VNNI kernel, gated CUDA DLL (#131, fixes #123)

2026-07-13 20:54:30 +02:00