dist_build() sorted the entire 151936-entry vocab by probability (qsort) on every
sampled token whenever 0 < g_nuc < 1 — the serve default — and again per draft
position under rejection sampling. Measured cost: 5.6-8.0 ms/call; the actual work
is finding the few-hundred-token head whose cumulative mass reaches g_nuc.
Replace the full qsort + linear scan with a max-heap partial select:
- Floyd heapify g_pidx over V by descending g_pbuf prob (O(V), cache-friendly)
- pop winners to the array's high end until cum >= g_nuc (k * O(log V))
- the remaining heap prefix IS the tail -> zero it, renormalize the head
Winners land in g_pidx[out..V-1] in descending order, so s2 accumulates in the
same order as before -> head is unchanged on tie-free shapes (ties were already
unspecified under the unstable qsort). All four dist_build/dist_sample contract
properties hold: g_pbuf stays id-indexed, g_pidx stays internal, the tail is
fully zeroed, the head renormalizes to 1.
No API change, no caller change, no new globals.
c/tests/test_topp.c (new): drives the REAL dist_build via the include-glm.c
pattern against an independent double-precision reimplementation of the OLD
algorithm. 123 cases: 6 sizes (1..1519) x 5 shapes (uniform/peaked/geometric/
plateau/sharptail) x 4 nuc values, plus the g_nuc>=1 guard-off paths and V=1.
Tie-free shapes compare head values within 1e-6 rel (float vs double renorm
noise); tie shapes compare value-multisets. No scratch files -> builds clean on
Windows MinGW without the unmerged mkdtemp shim (#352).
ENVIRONMENT.md says it is "generated from source by scanning every
getenv() site", but it had drifted far behind the code. Reconciling the
doc against the C sources (the MAINTAINING-DOCS.md procedure) found 34
engine env vars read by the code but undocumented, and nothing stale.
Added (defaults/effects taken from source, per the maintenance rules -
nothing invented):
Performance/tuning (11): COLI_NUMA, PILOT_TWO, COUPLE/COUPLE_K/COUPLE_D,
ROUTE_TRACE, COLI_NO_FUSED_PAIR, DISK_SPLIT, I4S, SPEC_PIN,
COLI_RAM_OVERCOMMIT
CUDA (16): COLI_CUDA_ATTN_SHARD, COLI_CUDA_PIPE/_PIPE_SHARD/_PIPE_S_MIN,
COLI_CUDA_MTP, COLI_CUDA_ASYNC, COLI_CUDA_DUAL_PROJ, COLI_CUDA_W4_PACKED,
COLI_CUDA_TC_INT4/_TC_MIN_ROWS/_TC_W4A16/_TC_W4A16_MIN,
COLI_CUDA_SHARED_W4A16/_SHARED_W4A16_MIN_ROWS, COLI_METAL_UNTRACKED
Advanced/debug (7): SCHEMA, EXPERT_BUDGET/_EXPERIMENTAL, TOKENS,
SCORE_PREFIX, REPIN_VERBOSE, PPL (olmoe-only), COLI_PROMPT (CLI section)
Also bumped the "Generated from" line to dev @ d5327e2 and noted the scan
now covers olmoe.c, backend_cuda.cu, backend_metal.mm (not just glm.c).
Verified: the code-vs-doc diff is now empty - all 111 distinct C env vars
are documented. The reverse diff (doc vars not in C code) is 14 entries,
all legitimate: 11 Python-side vars correctly in the Server/CLI section,
plus 3 prose constants (IOSQE_ASYNC, O_DIRECT, the VAR format word).
test_stops.c uses POSIX mkdtemp() to make a scratch dir in the CWD, but
MinGW-w64 does not declare mkdtemp, so the test failed to compile on the
Windows job - and only there:
tests/test_stops.c:73:9: error: implicit declaration of function 'mkdtemp';
did you mean 'mktemp'? [-Wimplicit-function-declaration]
make: *** [Makefile:318: tests/test_stops.exe] Error 1
That halted `make test` at the C-test stage on Windows (test_uring is
correctly Linux-only, so test_stops was the only blocker).
Added a compat_mkdtemp shim to compat.h, following the file's existing
convention (every platform difference lives there; the .c stays clean):
_mktemp fills the trailing X's in place (same contract as mkdtemp), then
_mkdir creates the directory. Also added <direct.h> for _mkdir. On Linux
compat.h is a complete no-op, so POSIX mkdtemp is untouched there.
Verified: test_stops builds clean on MinGW and all 5 sub-cases pass:
tokenizer special-flag parsing, config/generation_config eos union,
no-generation_config fallback, both-configs-mutilated tokenizer sweep,
and the T=NULL validation path.
A clean 'make glm' on MinGW emitted exactly two warnings, both real:
1. compat.h:240 - ignoring pragma comment [-Wunknown-pragmas]
#pragma comment(lib, "psapi.lib") is an MSVC directive; MinGW/GCC
warns about it. Guarded with ifdef _MSC_VER - MinGW links psapi via
-lpsapi (already in the Makefile), MSVC keeps the pragma.
2. glm.c:1210 - g_numa_nodes defined but not used [-Wunused-variable]
g_numa_nodes is only read/written inside ifdef __linux__ blocks, so on
every non-Linux build it is a static that is never used. Moved the
definition under the same __linux__ guard; nothing references it off-Linux.
Verified: rm -f *.o glm.exe && make glm -> 0 warnings, 0 errors.
The hot-thread tuning sets OMP_WAIT_POLICY=active and caps libgomp's spin
with GOMP_SPINCOUNT=200000. LLVM libomp reads NEITHER: under active policy it
sets KMP_BLOCKTIME=infinite, so once the answer ends and the engine parks on
stdin, the whole OMP team spins forever — ~100% per thread, the 3000% figure
reported on FreeBSD 15.1 in #341 (clang/libomp is the default toolchain
neighborhood there; the same applies to macOS builds).
KMP_BLOCKTIME=200 keeps the team hot for 200 ms after each parallel region —
plenty to bridge back-to-back per-expert matmuls — and then sleeps it.
setenv with overwrite=0, so a user-provided KMP_BLOCKTIME still wins, and
libgomp builds ignore the variable entirely: on GCC toolchains this commit
is a no-op by construction.
Not reproduced locally (this box is gcc/libgomp and idles clean); shipped as
the mechanism-matching candidate fix with a request on the issue for the
reporter to verify on dev and to name their OpenMP runtime (ldd | grep omp).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
No more manual pkill: cmd_serve writes a pidfile, cmd_stop finds the server
(pidfile, then /proc cmdline) and its engine (comm glm/exe/olmoe with SERVE=1
in environ — the engine re-execs for OMP tuning so its comm is 'exe', which is
why every 'pkill -x glm' in history killed nothing). SIGTERM, wait, SIGKILL.
--dry-run lists targets without acting. First real use took down a live
serve+engine pair cleanly and released 16 GB.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit 41a872c331a2a0a8655699e0171c68dd2bcda186)
The cold-chat cost, measured on this box: every `coli chat` spawns a private
engine (34-136 s of resident load) and starts with an empty expert cache (hit
4% cold vs 55% warm). Quitting throws both away.
`coli chat` now probes localhost:8000 first (~1 ms when nothing listens) and,
if a `coli serve` answers, runs the REPL over plain OpenAI SSE against it:
stdlib urllib only, engine byte-protocol untouched. --attach [URL] forces it,
--no-attach restores a private engine. reasoning_content keepalive pings are
filtered; :reset starts a new conversation client-side (the server's KV slots
reuse prefixes per conversation on their own).
Verified against a mock SSE server (pings ignored, markdown rendered, :reset,
clean exit) and against the real 744B model: two consecutive sessions, second
attach instant with zero reload — the engine stayed resident at 15.7 GB across
both. Honest limit: warmth carry-over BETWEEN different conversations is small
here because cap=3 slots/layer is a short memory; the structural wins are the
load never being repaid and same-conversation continuation.
LOCAL ONLY for now — not pushed, per the current working rule.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit aa406ccab8a4501b924bc3a9f4725dd1d18a685d)
Four bugs in the Nix flake, all addressed:
1. checkPhase failed with "make: python3: No such file or directory"
because `make test-c` shells out to python3 (c/Makefile: PYTHON ?=
python3) but python3 wasn't in nativeBuildInputs. Add pkgs.python3.
2. `coli serve` reported "engine is not built" even though glm was
packaged, because the engine landed in $out/bin while coli looks for
it beside itself, in libexec/colibri, or via $COLI_ENGINE (c/coli:57-
66). The wrapper now sets COLI_ENGINE to the bundled engine.
3. `coli serve` failed with ModuleNotFoundError: openai_server because
the support modules (openai_server.py, resource_plan.py, doctor.py)
were not installed. Install them and put their dir on PYTHONPATH.
4. Rebuilt the install layout into a self-contained $out/lib/colibri/
that mirrors the source tree coli expects, with $out/bin/{glm,coli}
as user-facing entry points (glm symlinked; coli wrapped).
NOTE: flake.lock is intentionally not included — it must be generated
on a Nix host via `nix flake lock` and committed separately, since it
requires narHash values that only the nix tooling can compute.
pipe_wait's sched_yield spin storms the scheduler for the full 0.5-3ms of
each in-flight expert read; behind COLI_PIPE_BLOCK=1 it parks on a condvar
instead (~5us wake, no lost-wakeup: workers store ready with release
BEFORE taking the mutex to broadcast, and the waiter re-checks under the
lock). Default OFF = byte-identical spin. Pthread pool only: the URING
backend has no waiter spin to replace.
Setting PIPE_WORKERS>0 in the env without PIPE now implies PIPE=1 with a
stderr note: sizing the pool declares the intent to use it (a full
benchmark campaign ran with PIPE_WORKERS=16 and the pipe silently off).
The implication fires only when the platform default left the pipe off
(no-op on _WIN32 where PIPE defaults to 1), only on a positive value
(PIPE_WORKERS=0/empty does not enable a clamped 1-worker pipe), and an
explicit PIPE=0 always wins. The rule lives in pipe_workers_imply_pipe()
so the table is unit-testable.
tests/test_pipe_block.c (in TEST_BINS, all platforms): pins the
implication table, and drives the pool through 200 generations under each
waiter against an on-disk expert fixture — spin as control, condvar arm
alternating parked and fast-path waits — verifying identical slot bytes.
Measured on the spin side (2x5090 + Gen5 NVMe, GLM-5.2 744B int4, CPU
decode): 1.98 -> 2.16 tok/s at 192 tokens (expert-disk service 44.6s ->
33.5s). On Metal/GPU decode an M5 Pro A/B (PR thread) measured no change,
consistent with the mechanism: the win is freeing the core the spinner
was stealing from the CPU matmul team.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The slow-filesystem warning fired for any model path under /mnt/, which
false-positives on native-Linux mounts (ZFS/ext4/xfs/NFS) that commonly
live there. Check the actual filesystem type via statfs() against the 9p
superblock magic (0x01021997) instead, so it only warns for a genuine
WSL 9p mount. Linux-only (statfs); no behavior change on other platforms.
Three vendor-neutral fixes to backend_cuda.cu, each with test coverage:
1. Upload check-order: a cached device tensor is now usable when the
caller's host pointers are stale or NULL. CUDA_RELEASE_HOST slots
null their host pointers after upload; the current engine reaches
those tensors through direct handles (coli_cuda_expert_mlp etc.), but
any caller going through coli_cuda_matmul/tensor_upload with a cached
tensor — as matmul_qt does for QT tensors — hits the !weights check
before the cached-tensor branch and fails spuriously. This hardens
the API contract rather than fixing a measured regression; the
contract is pinned by a 64x sustained-reuse test.
2. Sticky runtime error (real bug, test-caught): a failed allocation
left the last-error state set, so the NEXT healthy launch's
cudaGetLastError() check reported 'out of memory' and disabled a
perfectly good tensor. cuda_ok() now consumes the error on the
failure path; regression-covered.
3. COLI_GPU_FAIL_AFTER=N test hook: every GPU compute entry point (19
total: matmul, expert mlp/group, shared mlp, attention ops, pipe ops)
reports failure after N successful calls, so the engine's CPU
fallbacks and expert_host_ensure rematerialization can be exercised
end-to-end without real hardware faults. Unset = zero effect;
uploads/queries are never gated. Validated on GLM-5.2: total failure
(N=0) completes coherently with every released expert rematerialized.
Tests (run via make cuda-test on any CUDA GPU; vendor-neutral source):
64x sustained matmul reuse after host pointers are freed; upload from a
scribbled-and-freed temporary; five graceful upload-failure cases with
stats-integrity assertions; healthy-launch-after-failed-alloc (the
sticky-error regression); fault-hook on/off restore.
Verified on AMD RX 9070 XT via the companion HIP PR's compat header
(same test source); a make cuda-test run on NVIDIA hardware would
complete the matrix.
MinGW's UCRT has no getline; fixed-buffer fgets with CRLF trimming
reads the same case file everywhere.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same fix pattern as test_stops: mkdtemp with a CWD-relative template
(MinGW resolves Windows paths; /tmp is not one), and the fork/pipe/
truncate-based truncation subtest compiles only where those exist —
Windows still runs the cross-platform chunk-loop content check.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two latent bugs in every st.h reader, both hit in the field:
- a single pread caps at ~2^31 bytes on Linux, so any tensor past
2.1 GB (bf16 embed/unembed tensors of large models qualify) came
back silently truncated with perror printing '... : Success'
(errno untouched by a short read) — the same misleading-error
symptom glm.c fixed for its own reads in #236;
- no EINTR retry.
st_pread_full loops in ST_PREAD_CHUNK pieces (1 GB default, override
for tests), retries EINTR, and reports offset + progress on failure.
All five read sites converted; behavior on well-formed files is
byte-identical (GLM oracle re-verified on this branch: 32/32).
tests/test_st_pread builds with -DST_PREAD_CHUNK=7 so a 96-byte tensor
exercises the multi-chunk loop, and forks a child against a shard
truncated after st_init (init's static bounds check means the pread
path only fires when a file shrinks under a live handle) asserting
exit(1) with a 'short read' message and no 'Success'.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
tests/tok_o200k_tiny.json is a synthetic byte-level BPE (274 vocab, a
few KB) whose Split regex is the o200k pattern; expected ids in
tok_o200k_cases.txt were generated by HF tokenizers on that same file.
test_tok_o200k (in TEST_BINS) scores 40/40 encode + 40/40 round-trip:
case-transition splits, contractions, digit groups, the [\r\n/]* tail,
whitespace branches, CJK/Greek/Cyrillic, added-token atomicity.
The cl100k path is untouched by construction — dispatch requires
\p{Lu} in the tokenizer's own Split pattern, which cl100k lacks — and
stays covered by the GLM oracle (verified on this branch: 32/32).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Inkling ships an o200k-family tokenizer (case-aware Split regex, GPT-4o
lineage) rather than cl100k. tok_load now detects the family from the
pattern itself (\p{Lu} appears only in the o200k regex) so GLM behavior
is untouched, and encode dispatches to a new pretok_chunk_o200k that
replays the regex engine's backtracking order exactly: greedy optional
prefix, maximally-greedy uppercase run given back until the lowercase
run can match, contractions attached to letter runs, \p{N}{1,3}, and
the [\r\n/]* punctuation tail.
tok_unicode_o200k.h adds the two range tables the new classes need
(Lu+Lt and Lm+Lo+M), generated from Python unicodedata.
Validated against HF tokenizers on 357 adversarial strings (case
transitions, contractions, CJK, combining marks, emoji + modifiers,
zero-width chars, 300 mixed-charset fuzz cases): 357/357 identical.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
engine-cuda-syntax compiles backend_cuda.cu with nvcc's GCC host on Linux, and
check.yml's windows job is MinGW/UCRT64 CPU-only by design (#140). Neither
exercises nvcc's MSVC host — the only host compiler nvcc accepts on Windows — so
`make cuda-dll` and `make glm CUDA_DLL=1` are currently built by nothing.
The gap has already cost real bugs, all compile-time, all catchable without a GPU:
- #158: MSVC rejects the GCC-style -Xcompiler=-Wall,-Wextra ("D8021 invalid
numeric argument '/Wextra'"), CUDA_HOME/NVCC defaults were unresolvable, and
the kernel test used POSIX setenv. `make cuda-dll` was unbuildable as shipped.
- #314: CUDA_HOME containing spaces — the CUDA installer's default layout. This
job installs to exactly that path, so it reproduces the condition by default.
Two Windows-specific facts the job encodes, both verified by making it fail first:
- sub-packages needs "cudart", not just "nvcc": cuda-dll *links* the runtime, and
on Windows the headers/import lib are a separate installer component. With
'["nvcc"]' alone it dies at `#include <cuda_runtime.h>` — the Linux syntax job
never notices because it only compiles.
- runs-on is pinned to windows-2022: windows-latest now ships Visual Studio 18
(MSVC 14.5x) and CUDA's crt/host_config.h hard-errors on any host newer than
VS 2022. That is a real constraint on every Windows CUDA user today, so the pin
tracks what the toolkit supports rather than papering over it with
-allow-unsupported-compiler.
Build-only on purpose: hosted runners have no NVIDIA device, so this proves the
Windows+MSVC CUDA build stays buildable, not that the kernels or the DLL loader
behave on real silicon — that still needs hardware (#157). A check that overclaims
buys the false confidence the engine-cuda-syntax comment already warns about.
Verified green on a fork before proposing:
https://github.com/lEWFkRAD/colibri/actions/runs/29524302993
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
My test_stops.c called mkdtemp("/tmp/coli_stops_XXXXXX"). These binaries are
built by MinGW into NATIVE Windows .exe files, which resolve Windows paths —
"/tmp" is not one, so mkdtemp failed ENOENT and `make check` went red on the
windows job the moment #143 gave us cross-platform CI.
Now a relative template in the CWD, which is what test_compat_direct.c already
does (`#define TMPF "test_direct.tmp"`). test_uring.c uses /tmp but is
Linux-only by construction (Makefile guards it behind $(LINUX)); I copied the
wrong neighbour.
Worth being precise about what this was, because the red job looked scarier
than it was: Windows is fine. `make check` built the engine, ran the whole C
suite, and passed everything else — test_i4_grouped, kv_alloc, compat_direct —
before tripping on my temp path. The CI caught a bug in the test, not in the
port. That is #143 earning its keep on day one.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`(uintptr_t)p+n+4095 & ~(uintptr_t)4095` is CORRECT — `+` binds tighter than
`&` in C, so the rounding happens before the mask, which is what was meant.
But gcc -Wall warns (`suggest parentheses around '+' in operand of '&'`), and
this repo builds clean at -Wall -Wextra. Warning-free is a property worth more
than the two characters it costs to keep: it is what makes a NEW warning
visible instead of scrolling past in a wall of noise.
No behaviour change; the emitted code is identical.
Co-Authored-By: ZacharyZcR <#313>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@ZacharyZcR's workflow, already green on dev (78c77bf): engine + C suite,
web build + vitest, python suite, and a real CUDA syntax check.
Only the workflow file is cherry-picked here — main's engine code stays at
54cfe56 and nothing else from dev comes along. main is where the checks are
worth the most and where they were missing entirely.
Includes the two fixes the first run exposed: the pinned action's version
table stops at 12.6.2 (12.6.3 failed the install), and `nvcc ... | head -40`
took its exit status from head, so the job could never fail.
Co-Authored-By: ZacharyZcR <#144>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two independent defects in the CUDA syntax job, both caught on its first run.
1. `cuda: '12.6.3'` — Jimver/cuda-toolkit@v0.2.19's version table stops at
12.6.2, so the install step died with "Version not available: 12.6.3"
before nvcc was ever invoked. One digit.
2. `nvcc ... 2>&1 | head -40` — a pipeline exits with the status of its LAST
command, so head's 0 masked every nvcc error. The job printed "CUDA syntax
check passed" unconditionally: it could not fail. Defect 1 is why we found
out, since it broke the step *before* the pipe.
The second one is the one that matters. backend_cuda.cu is compiled by nothing
else in this repo — no local build, no test — so this job is the only thing
standing between a CUDA change and a user's GPU. A check that cannot fail is
worse than no check: it buys false confidence in exactly the file that most
needs the real thing. #298 spent a night debugging CUDA against no oracle at
all; this job is supposed to be that oracle.
It is also, precisely, the disease of the week in YAML form: a signal that
measures its own intention rather than the thing it claims to measure. See the
`~335 GB I/O saved` counter that counted dropped experts instead of bytes not
read (#303), and `route_agree: 95.3%` cited as "quality preserved" when it
only measures which experts coincide.
The other three jobs (engine, web, python) passed on the first run.
Co-Authored-By: ZacharyZcR <#144>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Feeds reference tokens through the normal step() decode path and reports
NLL/ppl + hit rate + tok/s + RSS. Inert unless PPL=1; default path
untouched (12/12 vs ref.json verified with patch applied). Cross-checked
vs HF transformers bf16 on identical token ids: 12.11 vs 12.25 ppl (#108).
GLM-5.2 MLA uses interleaved (DeepSeek-style) RoPE, which the C engine
implements. transformers < 5.11.0 applied split-half (Llama-style) RoPE in
GlmMoeDsa* instead; an oracle built on those versions silently drifts and the
engine scores 25/32 instead of the documented 32/32 (#281). Weights come out
identical across versions -- only the forward pass differs -- so a too-old
transformers produces an invalid ref_glm.json with no warning.
Add a version gate at the top of make_glm_oracle.py: hard sys.exit with an
actionable message citing the issue and the upgrade command. Reads the version
from importlib.metadata (authoritative installed-dist version) rather than the
mutable transformers.__version__ attribute -- the latter gets reset by the lazy
model-class import (from transformers import GlmMoeDsaForCausalLM), so reading
it after that import is unreliable. The gate runs before the heavy import and
falls back to the attribute only if the dist metadata lookup fails (editable/
src installs).
Validated end-to-end on transformers 5.13.1: script runs, ref_glm.json and
model.safetensors are byte-identical to the shipped versions, engine scores
32/32. With the floor raised to (5,14) the gate blocks with the expected
message.
The engine armed its stop tokens from config.json's eos_token_id and nothing
else. That trusts metadata written by third-party conversion tooling, which is
a thing we already know goes wrong: the README documents a mirror shipping
int4 MTP heads that silently give 0% draft acceptance. GLM-5.2 declares THREE
eos ids (<|endoftext|>, <|user|>, <|observation|>); a converter that rewrites
config.json with a reduced list leaves the engine stopping on fewer tokens
than the model emits, and the missed ones get detokenized and printed into the
chat as literal text while generation runs past the end of the turn.
Two independent defenses:
- eos_token_id is now unioned with generation_config.json, which is
HuggingFace's authority for generation (config.json often carries a
partial legacy copy). An extra stop is harmless; a missing one is not.
- every added-token the TOKENIZER marks "special":true is armed as a stop,
whatever the configs say. Those are control tokens (<|user|>, <|assistant|>,
<sop>, [gMASK], the image/video/audio markers) and are never legitimate
content in a reply -- GLM itself lists three of them as official eos.
<think>/<tool_call>/<arg_key> are "special":false and are deliberately NOT
swept up: they are real output. tok.h was parsing added_tokens but throwing
the "special" flag away, so the distinction wasn't available to anyone.
On the real per-row checkpoint this takes the armed set from 3 to 18:
[stop] 18 stop tokens: 154820 154827 154829 154821 ... (15 from the
tokenizer's special set)
Honesty about scope: this is hygiene for a class of bug, NOT a fix for the
trailing-junk report on #298 that prompted it. I hypothesised @woolcoxm's g64
checkpoint had lost eos ids in conversion; he checked, and it hadn't -- his
config arms all three correctly. The emit path is also innocent: is_stop() is
checked BEFORE emit() at every one of the four call sites (4215, 4256, 4908,
4987), so a correctly-armed stop cannot be printed. His trailing junk is still
unexplained and is more likely quantization noise. What this commit buys is
that a checkpoint we don't control cannot leak control tokens into a reply,
which was true before and is not now.
tests/test_stops.c covers both defenses: the union, a missing
generation_config.json, BOTH configs mutilated (the tokenizer still stops all
five control tokens while leaving <think> alone), and T=NULL (the validation
path keeps config-only behaviour).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@woolcoxm's matmul_i4_grouped_pair reads x once instead of twice for the
gate+up pair. Verified here against his branch (86e91b1 merged onto dev):
correct to ~2e-8 relative vs the double reference, and BIT-EXACT against two
separate matmul_i4_grouped calls on aligned shapes -- which is the shape the
real g64 checkpoints have (I = 2048 / 6144, gs = 64). His kernel is good.
Guarded behind COLI_HAVE_GROUPED_PAIR since the function only exists on that
branch; add -DCOLI_HAVE_GROUPED_PAIR to the test's Makefile rule when #298
lands and the pair cases activate.
The checks are deliberately asymmetric, and the reason is worth recording.
Bit-exactness is asserted ONLY when I % gs == 0: there every group is covered
by the AVX2 body, whose accumulation order matches the unfused kernel, so any
difference is a real bug. With a partial last group the tail falls to scalar
code and the compiler may contract/reassociate the fused body differently,
producing ~1e-7 differences -- rounding, not logic. My first version demanded
bit-exactness everywhere and duly "found" a bug in his kernel that did not
exist; the tell was that only `up` differed and never `gate`, which is FP luck
rather than a code path. Correctness is checked everywhere against the double
reference; identity only where identity is actually implied.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The coli CLI and openai_server.py need a real python3 interpreter, but a
fresh Windows resolves 'python' to the Microsoft Store alias stub (#198).
Document the winget one-liner as the interim fix suggested in #310 until
the CLI port lands.
matmul_i4_grouped is the reference the CUDA fmt=4 port (#298) is expected to
reproduce, and it had no test of its own. @woolcoxm is currently debugging a
CUDA backend against an oracle nobody had verified, which is two moving
targets at once -- and he can't cross-check on CPU, since a 5-prompt run
takes 8 hours on the 744B model.
This checks matmul_i4_grouped against a plain-C reference that dequantizes
nibble -> (v-8)*scale[i/gs] and accumulates in double, over 11 shapes: I a
clean multiple of gs, a partial last group (the glen clamp), odd I (the
scalar nibble tail), gs > I, gs=16/64/128, S>1, and the nibble extremes
0x00/0xFF -- which decode to -8/+7 because the format is offset-encoded, not
two's complement. Reading that backwards turns 15 into -1 and looks like
data-dependent noise rather than a bug.
All 11 shapes match to ~1e-8 relative, so the CPU kernel is exact and can be
trusted as the reference.
One note on the tolerance, because the first draft of this test got it wrong
and "found" a bug that wasn't there: the error is compared against the sum of
|terms|, not against |result|. A dot product of signed terms can land near
zero through cancellation, and then a 1e-6 absolute error -- ordinary f32
accumulator precision -- reads as a 1e-3 relative one. A wrong scale index or
a wrong group boundary shifts the result by a fraction of the terms, so it is
still caught at 1e-6.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Fix queue flush race: clear is_queued under mutex only; never move
pilot_w backwards (would break r<=w ring-buffer invariant). Worker
skips stale entries via new is_queued guard at start of pilot_realload
- Add early-exit in pilot_realload when is_queued==0 (entry flushed
between enqueue and worker pickup), preventing unnecessary loads
- Fix misleading EMA struct comment: momentum_logits is used only by
the PILOT prefetcher, not blended into actual MoE routing decisions
- Fix Slot pinned comment: 'never evicted' was too strong; clarify that
pinned slots may be displaced under extreme all-pinned cache pressure
- Use st_read_f32() for scale tensor (.qs) instead of st_read_raw() to
handle potential future BF16/F16 dtype changes robustly
woolcoxm