This website requires JavaScript.
d9d0e7b958
Merge pull request #288 from monotophic/fix/mux-1token
2026-07-15 22:00:32 +02:00
55576941cc
quant: generalize fmt=4 group-size detection (g64/g128/g256, not just 128)
2026-07-15 15:54:37 -04:00
a55cdfd4b8
cuda: gate pipe2 S-threshold on device count (single-GPU S=1, multi-GPU S>=8)
2026-07-15 15:36:10 -04:00
ca788833ab
feat(prefetch): implement persistent hot pinning and pre-warmup wait loop to break 94% hit rate barrier
2026-07-15 20:39:34 +02:00
24058d3de8
feat(pinning): implement dynamic asymmetric expert pinning and layer 0 EMA update to reach 90.9% hit rate and 3.25 tok/s
2026-07-15 20:27:27 +02:00
b3fdb145f8
feat(prefetch): add opt-in lookahead-3 prefetching option for PILOT=3
2026-07-15 20:17:45 +02:00
4ae4f61d16
feat(prefetch): implement stale request pruning and queue de-duplication to break 90% hit rate barrier
2026-07-15 20:15:19 +02:00
c4624276f3
feat(io): implement Consolidated Expert I/O to reduce expert disk reads by 3x and accelerate prefetching
2026-07-15 18:04:25 +02:00
ac1f7a8f38
feat(prefetch): implement prefetcher v2.1 with lookahead-2, hot pinning, adaptive cache, RMSNorm scaling, and routing EMA
2026-07-15 18:01:21 +02:00
043014536b
docs: CACHE_ROUTE breakthrough — 1.41 tok/s (4.3x over stock), misses 27%->17%
2026-07-15 13:40:51 -04:00
d0971ff7c1
cache: fix ws_b overcounting under EXPERT_BUDGET (cap 3->4, hit 57%->73%)
2026-07-15 13:16:53 -04:00
939b689a10
glm: hwinfo_emit Windows support — CPU brand, cores, RAM were hardcoded-zero
2026-07-15 12:14:13 -05:00
1a243cfc3e
coli: measured Windows launcher defaults — OMP tuning parity + DIRECT/PIPE/PILOT_REAL
2026-07-15 12:06:48 -05:00
4059e10761
docs(readme): mechanical fixes — expert count, IO_THREADS, line counts, desktop/
2026-07-15 01:21:15 -05:00
51dd4805af
benchmark_cuda_fixture: accept the current PROFILE service/wait format
2026-07-14 22:23:44 -05:00
f72ee6532d
resource_plan: count physical cores on Windows (GetLogicalProcessorInformationEx)
2026-07-14 22:05:38 -05:00
c5b1d14a74
cuda: enable pipe_layer_sparse resident pipeline at decode (S>=1 gate)
2026-07-15 12:25:16 -04:00
704b4d7eeb
Merge pull request #269 from michael-denyer/arm-i8mm-smmla
2026-07-15 16:20:32 +02:00
d0c9941a67
ARM i8mm: SMMLA fast path for the int8/int4 IDOT drivers
2026-07-15 14:59:06 +01:00
52550ff571
Merge pull request #267 from michael-denyer/fix-profile-disk-wait
2026-07-15 15:52:39 +02:00
da684b240d
profile: account expert-disk stall as wait, not other
2026-07-15 14:44:29 +01:00
dcc14217af
Merge pull request #259 from withinboredom/uring
2026-07-15 15:11:11 +02:00
3fdc6d394e
Merge remote-tracking branch 'origin/dev' into pr259
2026-07-15 15:09:42 +02:00
39a0777351
Merge pull request #261 from FECO-Admin/fix/randomuuid-fallback
2026-07-15 14:59:33 +02:00
0fc18abd24
Merge pull request #265 from skeldoor/feat/interrupt-generation
2026-07-15 14:59:24 +02:00
87ea13c963
Merge pull request #264 from skeldoor/perf/neon-multiacc-sdot
2026-07-15 14:59:12 +02:00
e871fb792d
Merge pull request #266 from skeldoor/fix/coli-tiers-leak
2026-07-15 14:57:08 +02:00
da260c38cb
server: clamp max_tokens to the server cap instead of rejecting it (fixes #260 )
2026-07-15 14:56:01 +02:00
b79610852b
coli: consume the engine's TIERS line at startup so it doesn't leak into the first answer
2026-07-15 13:37:36 +01:00
3745cc680e
glm.c/coli: Ctrl-C soft-stops the current turn instead of killing the engine
2026-07-15 13:37:02 +01:00
af421d6d3a
glm.c: 4-accumulator NEON SDOT for int8/int4 expert dots (2.4x/core on Apple Silicon)
2026-07-15 13:28:05 +01:00
869e80ee2d
web: fallback when crypto.randomUUID() is unavailable
2026-07-15 14:20:50 +02:00
5d4c3aa11b
Merge pull request #257 from woolcoxm/windows-optimizations
2026-07-15 13:42:23 +02:00
4d0438314a
Merge pull request #246 from woolcoxm/feat/diskio-kv-batching
2026-07-15 13:35:03 +02:00
a6895445a9
Merge pull request #254 from woolcoxm/feat/expert-budget
2026-07-15 13:34:19 +02:00
d9266e39e3
Merge pull request #248 from ZacharyZcR/serve/mux-kv-diag
2026-07-15 13:22:57 +02:00
d99d1a7a0e
Merge pull request #255 from ZacharyZcR/tools/quant-e8
2026-07-15 13:20:44 +02:00
e51ce378ca
Merge pull request #245 from ZacharyZcR/tools/atlas-unify
2026-07-15 13:20:37 +02:00
0b9ce242e4
fix wording
2026-07-15 04:07:24 +02:00
cf062010d8
use async and fix workers
2026-07-15 04:07:12 +02:00
5d1eb142ab
port uring implementation
2026-07-15 03:54:25 +02:00
25219de45b
windows: PIPE default ON + compat_fadvise WILLNEED cache-warmer (pread path)
2026-07-15 06:14:15 -04:00
0811730845
serve mux: keep ragged decode batches off the fused Metal kernels
2026-07-15 05:54:50 -04:00
0189a9f0ae
tools: E8 lattice quantization for the ablation harness — int{2,3,4,8}[-gN][-e8|-e8u][-rot] (#81 )
2026-07-15 17:24:16 +08:00
9c24b1eceb
Merge branch 'feat/expert-budget' into windows-dev
2026-07-15 03:23:25 -04:00
dd0d60692d
experiment: miss-aware EXPERT_BUDGET — keep all cache hits, only drop misses
2026-07-14 18:23:29 -04:00
413b370fbf
experiment: EXPERT_BUDGET=N — cap distinct experts/layer, up to 1.8x faster decode
2026-07-14 18:02:15 -04:00
71c262ce1a
diskio: KV write batching + persistent handle; generators: unfuse experts
2026-07-15 01:40:02 -04:00
e71d4fbe29
windows-dev: CUDA build script, expert budget + grouped quant research
2026-07-15 01:04:32 -04:00
2c6946c478
test-models: add --fp8 emission for the FP8->int4 converter test path
2026-07-15 01:02:21 -04:00
21eb86a0dd
diskio: research on disk I/O minimization for MoE expert streaming
2026-07-15 01:01:43 -04:00
69f65d5173
Windows-dev: grouped quantization + mixed precision + expert budget + download tool
2026-07-15 01:00:53 -04:00
e141db047d
converter: per-tensor-type mixed-precision control
2026-07-14 20:32:14 -04:00
d04d99e039
serve: print the KV prefix-reuse diagnostic in mux mode too — [API] KV slot line existed only in the legacy \x02PROMPT path (#153 )
2026-07-15 14:31:36 +08:00
e2ad6c72e6
diskio: KV write batching + persistent handle; generators: unfuse experts
2026-07-15 01:40:02 -04:00
3fd47b7bbd
Merge pull request #242 from woolcoxm/feat/grouped-quant-fmt4
2026-07-15 08:25:29 +02:00
e7cb7df501
tools: one expert atlas, not two — retire expert_atlas.py, analyze.py gains --web for the dashboard
2026-07-15 14:16:36 +08:00
ff04b320d0
security(win): load coli_cuda.dll by absolute path, never from the CWD (DLL hijack)
2026-07-15 08:16:20 +02:00
4268e00fa1
security: harden safetensors/JSON parsers against malicious model files
2026-07-15 08:13:20 +02:00
498ab0c20e
experiment: group-scaled int4 (fmt=4) — one scale per 128 elements, not per row
2026-07-14 18:57:35 -04:00
a8895f2d84
Merge pull request #193 from bokiko/feature/disk-split
2026-07-15 07:59:33 +02:00
c71e5d1cf8
glm: honest short-read reporting in expert pread; Makefile: -pthread for *BSD
2026-07-15 07:55:09 +02:00
ddf9212b81
glm: PIN_GB=all clamps to the --ram budget instead of pinning every expert (fixes #229 )
2026-07-15 07:50:48 +02:00
22c91403b7
Merge pull request #228 from bokiko/feature/score-prefix-default
2026-07-15 07:47:46 +02:00
00902a2ea9
Merge pull request #224 from michael-denyer/fix-kv-alloc-double-free
2026-07-15 07:47:16 +02:00
5025f00428
Merge pull request #194 from bokiko/feature/glm-prefix-default
2026-07-15 07:46:47 +02:00
a28f31fa3b
tools: expert_atlas — confound-controlled probe harness for the GLM-5.2 expert atlas (#175 ) (#218 )
2026-07-15 07:41:24 +02:00
416570339f
tests: remove accidentally committed Linux test binaries + gitignore them — fixes make test-c on non-Linux (#226 )
2026-07-15 06:40:42 +01:00
2bdf054622
glm: fix kv_alloc double-free on KV re-allocation (stale pre-Metal free block)
2026-07-15 00:33:45 +01:00
69004f73c2
tests: kv_alloc re-allocation regression test
2026-07-15 00:32:58 +01:00
62419af188
routing: opt-in CACHE_ROUTE max-rank prefer (pin∪LRU), default off — 2.4->3.33 tok/s on GB10 (#199 )
2026-07-14 20:08:28 +00:00
4a936c8af3
FreeBSD compatibility: platform guards in glm.c/olmoe.c (#206 )
2026-07-14 13:08:12 -07:00
bfd49c41b9
Makefile: select $(PYTHON) from the host, not the target triple (#205 )
2026-07-14 22:07:56 +02:00
8c18b7af68
openai_server: fix path traversal in web static serving (relative_to, not startswith) + empty-prompt validation, with tests (#212 )
2026-07-14 23:07:41 +03:00
bad64d1b06
Makefile: opt-in NVCC_CCBIN=<compiler> to override nvcc's host compiler (fixes #211 )
2026-07-14 20:44:41 +02:00
6ca9565d2d
pilot: two-step shared-expert router prediction, +3.1% recall, opt-in PILOT_TWO=1 (#204 , #200 )
2026-07-14 14:07:51 -04:00
ab09dc52f3
bench: pass the resolved GLM engine path to eval_glm.py — cwd-relative ./glm default broke coli bench from other dirs (#203 )
2026-07-14 19:55:36 +03:00
e21318e678
README: dashboard screenshots + contents index + Web dashboard section (#202 )
2026-07-15 00:45:42 +08:00
21cbf14d59
cuda(win): export + load the #111 pipeline ABI — 24 missing DLLEXPORT decorations + loader wrappers, unbreaks CUDA_DLL=1 (#201 )
2026-07-14 09:43:56 -07:00
3045b35cdd
bench: default --data to the XDG/LocalAppData cache dir instead of the repo tree (#190 )
2026-07-14 19:18:49 +03:00
ec89136029
GPU resident pipeline: batch CUDA attention, head-sharded kv_b, prefill expert groups, W4A16 mixed dispatch (#111 )
2026-07-15 00:18:05 +08:00
2ead86a27f
serve mux: fix Windows request dispatch — BINARY-mode stdin/stdout + PeekNamedPipe-only polling (fixes #195 )
2026-07-14 18:15:29 +02:00
e12a4295cc
glm.c: silence unused-variable 'c' in hwinfo_emit (reported in #148 on Darwin, present on Linux too)
2026-07-14 18:03:56 +02:00
372f8fd633
run_tests: normpath test binaries — forward-slash relative paths fail CreateProcess on Windows (#196 )
2026-07-14 09:01:12 -07:00
4a0435cbb3
glm.c: remove the LMHEAD_EXACT probe accidentally merged with #152 — functional no-op strip (#197 )
2026-07-14 18:01:04 +02:00
c4d14017df
engine: default the [gMASK]<sop> prefix ON in SCORE mode for GLM snapshots (#194 , #108 )
2026-07-14 18:28:41 +03:00
f7c7ab6bde
ci: run make check on ubuntu / windows (MSYS2 UCRT64) / macos (#140 )
2026-07-13 15:38:40 -04:00
5230717cf4
eval: default the [gMASK]<sop> prefix ON for GLM snapshots (#108 )
2026-07-14 15:19:48 +03:00
b611304166
stats: DISK_SPLIT=1 opt-in disk-load split by speculation context and layer kind
2026-07-14 15:17:08 +03:00
f1fa5bf3c8
placement: full-resident experts on large-memory hosts — CUDA_EXPERT_GB=auto, PIN_GB=all, adaptive GPU slots, RoPE cache (#80 )
2026-07-14 22:35:39 +08:00
504fcdd930
SCHEMA=<file.json>: JSON-Schema -> GBNF compiler for grammar-forced drafts (#148 )
2026-07-14 15:31:09 +01:00
9b08cbc543
cuda-dll: make the Windows CUDA build work as shipped — MSVC host flags, CUDA_PATH defaults, POSIX shim in kernel test (#158 , #157 )
2026-07-14 10:28:36 -04:00
12350a494f
prefill: batch the attention input projections — bit-identical, -4.5% prefill (#152 )
2026-07-14 16:26:07 +02:00
37ffb61674
routing: cross-layer coupling trace dump + pair-table builder + opt-in coupled prefetch source (#176 )
2026-07-14 15:13:19 +01:00
2878c60987
web dashboard: live expert-tier panel (VRAM/RAM/disk), one-process hosting, coli web (#126 )
2026-07-14 22:12:53 +08:00
d47b095875
Windows: direct I/O (FILE_FLAG_NO_BUFFERING, 1.47x decode) + VirtualLock pin wiring + select_ctx device cache (#162 )
2026-07-14 07:12:15 -07:00
ec0aadf91a
olmoe: int8 container support (fixes SIGBUS), NEON int8 dot kernels (2.7x on ARM), macOS RSS fix, converter fallback (#187 )
2026-07-14 23:59:49 +10:00
32629eae7f
serve: WaitForSingleObject/PeekNamedPipe stdin polling for run_serve_mux on native Windows (#177 , fixes #189/#139)
2026-07-14 09:44:37 -04:00
d872dd5b4e
glm.c: harden 5 crash paths — qk_rope buffer guard, OOM-checked mallocs, null derefs (#183 )
2026-07-14 09:43:55 -04:00
0f5d5c8541
loaders: fix silent short-read (uninitialized memory) in config/oracle readers (#181 )
2026-07-14 09:43:20 -04:00
Vincenzo