`coli serve` runs the engine with SERVE_BATCH=1 (openai_server.py), which
selects run_serve_mux, which sets g_draft=0 — speculation isn't ragged-safe
across the multi-slot batch. But the "[MTP] active: native speculative decoding
(draft=N)" line is printed in main() BEFORE the serve path is chosen, so
`coli serve DRAFT=8` announced draft=8 and then silently disabled it.
@LordMZTE reported the misleading message.
The load line and the [MTP] stderr line now detect the mux case (SERVE +
SERVE_BATCH) and report it truthfully: "MTP DISABLED (multiplexed serve)" with
draft=0, plus a one-line explanation that single-client interactive use
(`coli chat`, which spawns run_serve without SERVE_BATCH) keeps MTP. No more
draft=8 claim on a path that runs draft=0.
This is the honesty half of #358. The feature half — MTP inside the HTTP
server for a single client — is a real enhancement but needs engine work: the
mux decode kernel would have to run the speculative path when exactly one KV
slot is active (S=1 is not actually ragged), and it needs a server round-trip
test. Tracked separately; the message no longer lies in the meantime.
Reported-by: LordMZTE <#358>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The expert lookup counted a hit identically whether the pinned hot-store or the
LRU ecache served it — both Phase C branches bumped the single `m->hits`. With a
warm pin profile the pin tier absorbs most hot experts, so the ecache could be
serving anywhere from ~0% to most hits and the logs couldn't tell which. That
made every cache-policy question unanswerable, including #223's "at what cap
does the eviction policy start to win?" (a flat A/B can mean "pin absorbed
everything" or "genuine floor" and the lumped counter can't distinguish them).
Two counters `hit_pin`/`hit_ecache` bumped at the two lookup branches (the
existing `m->hits++` stays, so all existing math is unchanged), snapshotted in
ProfBase and reset alongside `hits`. Surfaced in the human-readable summaries:
decode: expert hit rate 31.3% (pin 22.1% + lru 9.2%)
[PROF]: hit 31.3% (X pin + Y lru / Z load)
tiny: hit rate 88.1% (0 pin + 74 lru / 10 miss)
The serve-mux STAT protocol line is untouched (openai_server.py parses it
positionally).
Invariant hit_pin + hit_ecache == hits holds by construction — exactly two
sites bump hits and each also bumps one split counter, nothing else mutates
hits. Verified numerically on the tiny oracle: 0 pin + 74 lru = 74 hits, 88.1%.
Zero cost: two increments on paths that already increment.
Reported-by: KingIcyCreamProjects <#336>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The local (`--indir`) branch of main() ignored --mtp and --indexer entirely:
it always called convert_shard() without keep_mtp/keep_idx and always wrote
`out-NNNNN.safetensors`. So the documented two-pass workflow — convert the main
model into an outdir, then run `--mtp` into the SAME outdir for the head — did
the opposite on the local path: with --mtp defaulting ebits to 8 and keep_mtp
staying False, it silently re-converted the whole model to per-row int8 and
overwrote the finished fmt=4 container shard by shard, printing nothing wrong.
@mohamedmastouri2000-boop lost 137 of 141 freshly-converted g64 shards to this
while building the public container for #298/#326.
The --indir branch now mirrors the download path: keep_mtp=a.mtp /
keep_idx=a.indexer passed through, output named out-mtp-/out-idx-/out- by mode,
empty shards skipped (an MTP pass emits only shards containing layer n_layers),
and config/tokenizer copied only on the main pass (the head/idx passes land in
an already-complete outdir).
Verified with a synthetic 2-layer + MTP-shard model: after the main pass, a
second --mtp pass into the same outdir leaves out-00000's md5 BYTE-IDENTICAL
and writes out-mtp-00000 alongside it. The bug would have changed that md5.
Reported-by: mohamedmastouri2000-boop <#355>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A single NaN or +Inf logit silently broke the default sampling path. +Inf
became `mx`, then `expf((Inf-mx))`/`expf((NaN-mx))` is NaN, the softmax sum went
NaN, every probability went NaN — and dist_sample's fallback loop
`if(g_pbuf[i]>0)` is false for NaN at every index, so it returned 0. Every
subsequent token: 0. No error, no warning. @KingIcyCreamProjects found it.
dist_build now takes `mx` over finite logits only, gives a non-finite logit
probability 0, and when the distribution is unusable (no finite logit, or a
non-finite/zero sum) collapses to a delta on the finite argmax and warns ONCE
on stderr — degraded, but a valid token and a visible cause, never a silent
stream of zeros. The finite argmax uses the index found during the mx pass
(robust even when lo[0] itself is NaN, where argmax_v would wrongly return 0).
tests/test_sample_nan.c: healthy logits still sample correctly; NaN/+Inf
injected at lo[0], the middle, and the last position all pick the finite
argmax; an all-non-finite vocab leaves no NaN in the buffer and doesn't crash.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
An all-grouped container (kv_b_proj at fmt=4) generates one correct token
and then EOS: qt_addrow and qt_matvec_rows handle fmt 0/1/2 and fall
through to the int2 decoder, so grouped-int4 kv_b was unpacked as 2-bit
pairs under a per-row scale that does not exist in the [O,ng] layout.
Prefill (S>4, reconstruction) is unaffected, which made the failure look
like an EOS bug rather than an attention bug.
Same class as #298 (CUDA absorb kernels missing fmt=4), CPU side. Existing
containers escape it because the recommended mixed-precision recipe keeps
kv_b at int8 (#237).
Adds per-group branches mirroring matmul_i4_grouped semantics. fmt 0/1/2/3
paths are untouched.
Co-Authored-By: Claude <noreply@anthropic.com>
compat_pread mapped every non-EOF ReadFile failure to EIO, so the field report
was always the same three words: "Input/output error". #307 then burned three
rounds of guessing across three people — storage? contention? alignment? —
because the actual GetLastError code never appeared anywhere.
compat_pread now stashes the code in a per-thread slot and pread_full appends
it to the failure line:
pread qs: Input/output error (off 780592, 0/8192 bytes, WinErr=1450)
That one number is the difference between "insufficient system resources"
(1450 -> memory pressure), "sharing violation" (32 -> AV interference),
"device not ready" (21 -> the T: drive itself) and a dozen other distinct
diagnoses. Diagnostic only: no behavior change, no retry policy — that
discussion lives in #361 and should be settled AFTER the first report tells
us which error we are actually retrying.
Linux path untouched byte-for-byte; the Windows compile is certified by the
check.yml MSYS2 job.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The DSA lightning indexer selects the top-index_topk (2048) context keys to
attend to by finding the threshold = keep-th largest attention score. It
previously full-qsorted all nk scores per layer per token (O(nk log nk)) just
to read one pivot value, then scanned the original array in position order to
build the kept set.
Replace the qsort with partial_select_desc (Hoare quickselect, median-of-three,
descending): O(nk) average to partition the keep largest into a[0..k), then the
threshold is min of that block. The two position-order scans (>thr then ==thr)
are UNCHANGED, so the kept-position set is bit-identical -- a stronger contract
than #335's sampling heap (which was multiset-only because the heap was unstable
and changed accumulation order). The quickselect pivot IS by definition the
keep-th largest, so the new threshold equals old tmp[keep-1] exactly.
Measured (bench_dsa_select, keep=2048, median of 2000 reps):
nk=2049: 119us -> 5.7us (21x)
nk=8192: 626us -> 43us (15x)
nk=32768: 2.8ms -> 0.28ms (10x)
nk=65536: 6.6ms -> 0.47ms (14x)
The gap widens with context (linear vs n-log-n). DSA only fires past index_topk,
so this is precisely the long-conversation regime where decode latency matters.
Adds test_dsa_select (in TEST_BINS): 129 cases asserting element-wise identical
kept-set vs an independent qsort reference across shapes (random, peaked,
sorted, reverse-sorted, tie-plateau, all-equal) and edges (keep==1, keep==nk).
Also directly checks the partition invariant.
Adds bench_dsa_select (on-demand, NOT a gate): reproduces the table above.
test_topp proves correctness; bench_topp measures the latency claim. It re-implements
the OLD dist_build (full-vocab qsort) inline on a private buffer and times it against
the REAL new dist_build over identical inputs in one process: V=151936, temp=0.7,
3 shapes (realistic / uniform / plateau) x 4 nuc values (0.5/0.9/0.95/0.99), 2000
timed reps each, median reported. Deliberately NOT in TEST_BINS -- it's a microbench,
not a gate. Build on demand: make tests/bench_topp && ./tests/bench_topp
dist_build() sorted the entire 151936-entry vocab by probability (qsort) on every
sampled token whenever 0 < g_nuc < 1 — the serve default — and again per draft
position under rejection sampling. Measured cost: 5.6-8.0 ms/call; the actual work
is finding the few-hundred-token head whose cumulative mass reaches g_nuc.
Replace the full qsort + linear scan with a max-heap partial select:
- Floyd heapify g_pidx over V by descending g_pbuf prob (O(V), cache-friendly)
- pop winners to the array's high end until cum >= g_nuc (k * O(log V))
- the remaining heap prefix IS the tail -> zero it, renormalize the head
Winners land in g_pidx[out..V-1] in descending order, so s2 accumulates in the
same order as before -> head is unchanged on tie-free shapes (ties were already
unspecified under the unstable qsort). All four dist_build/dist_sample contract
properties hold: g_pbuf stays id-indexed, g_pidx stays internal, the tail is
fully zeroed, the head renormalizes to 1.
No API change, no caller change, no new globals.
c/tests/test_topp.c (new): drives the REAL dist_build via the include-glm.c
pattern against an independent double-precision reimplementation of the OLD
algorithm. 123 cases: 6 sizes (1..1519) x 5 shapes (uniform/peaked/geometric/
plateau/sharptail) x 4 nuc values, plus the g_nuc>=1 guard-off paths and V=1.
Tie-free shapes compare head values within 1e-6 rel (float vs double renorm
noise); tie shapes compare value-multisets. No scratch files -> builds clean on
Windows MinGW without the unmerged mkdtemp shim (#352).
A clean 'make glm' on MinGW emitted exactly two warnings, both real:
1. compat.h:240 - ignoring pragma comment [-Wunknown-pragmas]
#pragma comment(lib, "psapi.lib") is an MSVC directive; MinGW/GCC
warns about it. Guarded with ifdef _MSC_VER - MinGW links psapi via
-lpsapi (already in the Makefile), MSVC keeps the pragma.
2. glm.c:1210 - g_numa_nodes defined but not used [-Wunused-variable]
g_numa_nodes is only read/written inside ifdef __linux__ blocks, so on
every non-Linux build it is a static that is never used. Moved the
definition under the same __linux__ guard; nothing references it off-Linux.
Verified: rm -f *.o glm.exe && make glm -> 0 warnings, 0 errors.
The hot-thread tuning sets OMP_WAIT_POLICY=active and caps libgomp's spin
with GOMP_SPINCOUNT=200000. LLVM libomp reads NEITHER: under active policy it
sets KMP_BLOCKTIME=infinite, so once the answer ends and the engine parks on
stdin, the whole OMP team spins forever — ~100% per thread, the 3000% figure
reported on FreeBSD 15.1 in #341 (clang/libomp is the default toolchain
neighborhood there; the same applies to macOS builds).
KMP_BLOCKTIME=200 keeps the team hot for 200 ms after each parallel region —
plenty to bridge back-to-back per-expert matmuls — and then sleeps it.
setenv with overwrite=0, so a user-provided KMP_BLOCKTIME still wins, and
libgomp builds ignore the variable entirely: on GCC toolchains this commit
is a no-op by construction.
Not reproduced locally (this box is gcc/libgomp and idles clean); shipped as
the mechanism-matching candidate fix with a request on the issue for the
reporter to verify on dev and to name their OpenMP runtime (ldd | grep omp).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
No more manual pkill: cmd_serve writes a pidfile, cmd_stop finds the server
(pidfile, then /proc cmdline) and its engine (comm glm/exe/olmoe with SERVE=1
in environ — the engine re-execs for OMP tuning so its comm is 'exe', which is
why every 'pkill -x glm' in history killed nothing). SIGTERM, wait, SIGKILL.
--dry-run lists targets without acting. First real use took down a live
serve+engine pair cleanly and released 16 GB.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit 41a872c331a2a0a8655699e0171c68dd2bcda186)
The cold-chat cost, measured on this box: every `coli chat` spawns a private
engine (34-136 s of resident load) and starts with an empty expert cache (hit
4% cold vs 55% warm). Quitting throws both away.
`coli chat` now probes localhost:8000 first (~1 ms when nothing listens) and,
if a `coli serve` answers, runs the REPL over plain OpenAI SSE against it:
stdlib urllib only, engine byte-protocol untouched. --attach [URL] forces it,
--no-attach restores a private engine. reasoning_content keepalive pings are
filtered; :reset starts a new conversation client-side (the server's KV slots
reuse prefixes per conversation on their own).
Verified against a mock SSE server (pings ignored, markdown rendered, :reset,
clean exit) and against the real 744B model: two consecutive sessions, second
attach instant with zero reload — the engine stayed resident at 15.7 GB across
both. Honest limit: warmth carry-over BETWEEN different conversations is small
here because cap=3 slots/layer is a short memory; the structural wins are the
load never being repaid and same-conversation continuation.
LOCAL ONLY for now — not pushed, per the current working rule.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit aa406ccab8a4501b924bc3a9f4725dd1d18a685d)
The slow-filesystem warning fired for any model path under /mnt/, which
false-positives on native-Linux mounts (ZFS/ext4/xfs/NFS) that commonly
live there. Check the actual filesystem type via statfs() against the 9p
superblock magic (0x01021997) instead, so it only warns for a genuine
WSL 9p mount. Linux-only (statfs); no behavior change on other platforms.
Same fix pattern as test_stops: mkdtemp with a CWD-relative template
(MinGW resolves Windows paths; /tmp is not one), and the fork/pipe/
truncate-based truncation subtest compiles only where those exist —
Windows still runs the cross-platform chunk-loop content check.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two latent bugs in every st.h reader, both hit in the field:
- a single pread caps at ~2^31 bytes on Linux, so any tensor past
2.1 GB (bf16 embed/unembed tensors of large models qualify) came
back silently truncated with perror printing '... : Success'
(errno untouched by a short read) — the same misleading-error
symptom glm.c fixed for its own reads in #236;
- no EINTR retry.
st_pread_full loops in ST_PREAD_CHUNK pieces (1 GB default, override
for tests), retries EINTR, and reports offset + progress on failure.
All five read sites converted; behavior on well-formed files is
byte-identical (GLM oracle re-verified on this branch: 32/32).
tests/test_st_pread builds with -DST_PREAD_CHUNK=7 so a 96-byte tensor
exercises the multi-chunk loop, and forks a child against a shard
truncated after st_init (init's static bounds check means the pread
path only fires when a file shrinks under a live handle) asserting
exit(1) with a 'short read' message and no 'Success'.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
My test_stops.c called mkdtemp("/tmp/coli_stops_XXXXXX"). These binaries are
built by MinGW into NATIVE Windows .exe files, which resolve Windows paths —
"/tmp" is not one, so mkdtemp failed ENOENT and `make check` went red on the
windows job the moment #143 gave us cross-platform CI.
Now a relative template in the CWD, which is what test_compat_direct.c already
does (`#define TMPF "test_direct.tmp"`). test_uring.c uses /tmp but is
Linux-only by construction (Makefile guards it behind $(LINUX)); I copied the
wrong neighbour.
Worth being precise about what this was, because the red job looked scarier
than it was: Windows is fine. `make check` built the engine, ran the whole C
suite, and passed everything else — test_i4_grouped, kv_alloc, compat_direct —
before tripping on my temp path. The CI caught a bug in the test, not in the
port. That is #143 earning its keep on day one.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`(uintptr_t)p+n+4095 & ~(uintptr_t)4095` is CORRECT — `+` binds tighter than
`&` in C, so the rounding happens before the mask, which is what was meant.
But gcc -Wall warns (`suggest parentheses around '+' in operand of '&'`), and
this repo builds clean at -Wall -Wextra. Warning-free is a property worth more
than the two characters it costs to keep: it is what makes a NEW warning
visible instead of scrolling past in a wall of noise.
No behaviour change; the emitted code is identical.
Co-Authored-By: ZacharyZcR <#313>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Feeds reference tokens through the normal step() decode path and reports
NLL/ppl + hit rate + tok/s + RSS. Inert unless PPL=1; default path
untouched (12/12 vs ref.json verified with patch applied). Cross-checked
vs HF transformers bf16 on identical token ids: 12.11 vs 12.25 ppl (#108).
GLM-5.2 MLA uses interleaved (DeepSeek-style) RoPE, which the C engine
implements. transformers < 5.11.0 applied split-half (Llama-style) RoPE in
GlmMoeDsa* instead; an oracle built on those versions silently drifts and the
engine scores 25/32 instead of the documented 32/32 (#281). Weights come out
identical across versions -- only the forward pass differs -- so a too-old
transformers produces an invalid ref_glm.json with no warning.
Add a version gate at the top of make_glm_oracle.py: hard sys.exit with an
actionable message citing the issue and the upgrade command. Reads the version
from importlib.metadata (authoritative installed-dist version) rather than the
mutable transformers.__version__ attribute -- the latter gets reset by the lazy
model-class import (from transformers import GlmMoeDsaForCausalLM), so reading
it after that import is unreliable. The gate runs before the heavy import and
falls back to the attribute only if the dist metadata lookup fails (editable/
src installs).
Validated end-to-end on transformers 5.13.1: script runs, ref_glm.json and
model.safetensors are byte-identical to the shipped versions, engine scores
32/32. With the floor raised to (5,14) the gate blocks with the expected
message.
The engine armed its stop tokens from config.json's eos_token_id and nothing
else. That trusts metadata written by third-party conversion tooling, which is
a thing we already know goes wrong: the README documents a mirror shipping
int4 MTP heads that silently give 0% draft acceptance. GLM-5.2 declares THREE
eos ids (<|endoftext|>, <|user|>, <|observation|>); a converter that rewrites
config.json with a reduced list leaves the engine stopping on fewer tokens
than the model emits, and the missed ones get detokenized and printed into the
chat as literal text while generation runs past the end of the turn.
Two independent defenses:
- eos_token_id is now unioned with generation_config.json, which is
HuggingFace's authority for generation (config.json often carries a
partial legacy copy). An extra stop is harmless; a missing one is not.
- every added-token the TOKENIZER marks "special":true is armed as a stop,
whatever the configs say. Those are control tokens (<|user|>, <|assistant|>,
<sop>, [gMASK], the image/video/audio markers) and are never legitimate
content in a reply -- GLM itself lists three of them as official eos.
<think>/<tool_call>/<arg_key> are "special":false and are deliberately NOT
swept up: they are real output. tok.h was parsing added_tokens but throwing
the "special" flag away, so the distinction wasn't available to anyone.
On the real per-row checkpoint this takes the armed set from 3 to 18:
[stop] 18 stop tokens: 154820 154827 154829 154821 ... (15 from the
tokenizer's special set)
Honesty about scope: this is hygiene for a class of bug, NOT a fix for the
trailing-junk report on #298 that prompted it. I hypothesised @woolcoxm's g64
checkpoint had lost eos ids in conversion; he checked, and it hadn't -- his
config arms all three correctly. The emit path is also innocent: is_stop() is
checked BEFORE emit() at every one of the four call sites (4215, 4256, 4908,
4987), so a correctly-armed stop cannot be printed. His trailing junk is still
unexplained and is more likely quantization noise. What this commit buys is
that a checkpoint we don't control cannot leak control tokens into a reply,
which was true before and is not now.
tests/test_stops.c covers both defenses: the union, a missing
generation_config.json, BOTH configs mutilated (the tokenizer still stops all
five control tokens while leaving <think> alone), and T=NULL (the validation
path keeps config-only behaviour).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@woolcoxm's matmul_i4_grouped_pair reads x once instead of twice for the
gate+up pair. Verified here against his branch (86e91b1 merged onto dev):
correct to ~2e-8 relative vs the double reference, and BIT-EXACT against two
separate matmul_i4_grouped calls on aligned shapes -- which is the shape the
real g64 checkpoints have (I = 2048 / 6144, gs = 64). His kernel is good.
Guarded behind COLI_HAVE_GROUPED_PAIR since the function only exists on that
branch; add -DCOLI_HAVE_GROUPED_PAIR to the test's Makefile rule when #298
lands and the pair cases activate.
The checks are deliberately asymmetric, and the reason is worth recording.
Bit-exactness is asserted ONLY when I % gs == 0: there every group is covered
by the AVX2 body, whose accumulation order matches the unfused kernel, so any
difference is a real bug. With a partial last group the tail falls to scalar
code and the compiler may contract/reassociate the fused body differently,
producing ~1e-7 differences -- rounding, not logic. My first version demanded
bit-exactness everywhere and duly "found" a bug in his kernel that did not
exist; the tell was that only `up` differed and never `gate`, which is FP luck
rather than a code path. Correctness is checked everywhere against the double
reference; identity only where identity is actually implied.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
matmul_i4_grouped is the reference the CUDA fmt=4 port (#298) is expected to
reproduce, and it had no test of its own. @woolcoxm is currently debugging a
CUDA backend against an oracle nobody had verified, which is two moving
targets at once -- and he can't cross-check on CPU, since a 5-prompt run
takes 8 hours on the 744B model.
This checks matmul_i4_grouped against a plain-C reference that dequantizes
nibble -> (v-8)*scale[i/gs] and accumulates in double, over 11 shapes: I a
clean multiple of gs, a partial last group (the glen clamp), odd I (the
scalar nibble tail), gs > I, gs=16/64/128, S>1, and the nibble extremes
0x00/0xFF -- which decode to -8/+7 because the format is offset-encoded, not
two's complement. Reading that backwards turns 15 into -1 and looks like
data-dependent noise rather than a bug.
All 11 shapes match to ~1e-8 relative, so the CPU kernel is exact and can be
trusted as the reference.
One note on the tolerance, because the first draft of this test got it wrong
and "found" a bug that wasn't there: the error is compared against the sum of
|terms|, not against |result|. A dot product of signed terms can land near
zero through cancellation, and then a 1e-6 absolute error -- ordinary f32
accumulator precision -- reads as a 1e-3 relative one. A wrong scale index or
a wrong group boundary shifts the result by a fraction of the terms, so it is
still caught at 1e-6.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
make_glm_oracle.py wrote glm_tiny/model.safetensors and glm_tiny/config.json
without creating the directory first. safetensors.save_file writes a temp file
inside the target dir before the atomic rename, so on a clean checkout (no
pre-existing glm_tiny/) it aborts with an opaque error:
SafetensorError: Error while serializing: I/O error:
The system cannot find the path specified. (os error 3)
at path ".../glm_tiny/.tmpXXXXXX"
The directory only ever existed because it was left over from a previous run,
so the first-ever `python tools/make_glm_oracle.py` fails for every new user
following the README's verify step.
Create glm_tiny/ with Path.mkdir(parents=True, exist_ok=True) before the save
branch — covers the fp8 path, the bf16 path, and config.json. Path is already
imported; no new dependency, no change to the CPU build.
The dashboard's wall-time stack assumed ewait = felt I/O stall and
edisk = overlapped read service, but the engine measured neither:
t_ewait was declared and reported yet never incremented (the blue
'I/O wait' segment always read 0.00s), while t_edisk accumulated on
the COMPUTE thread — blocking OMP loads, PIPE dispatch, pipe_wait
spins — i.e. exactly the felt stall. The UI excluded it from the
stack as 'overlapped', so the whole disk stall landed in 'Other'
(e.g. 46.9s Other vs 45.3s disk on a 54.1s turn).
Now the semantics match the contract:
- t_ewait accumulates the compute-thread stall (blocking loads,
PIPE dispatch, pipe_wait spins) and feeds the in-stack I/O wait.
- Disk service is real: expert_load is timed on whichever thread
runs the read (PIPE workers, OMP loaders, pilot) into an atomic
ns counter (g_edisk_ns) — thread-seconds, overlapped with compute.
- prof_report's I/O share and verdict use the felt wait only, and
the expert I/O line prints 'read service / felt wait' so the
wait:service gap shows how well PIPE is hiding the reads.
PROF wire format, /profile JSON and the web UI are unchanged —
the dashboard's existing assumptions are simply true now.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dd4hHD2Dsvgr7VAwEVFqWK
The engine already tracks where each turn's wall time goes (expert-disk
service, I/O wait, expert matmul, attention, lm_head) — it just only spoke
at exit or under PROF=1. Stream it instead:
- glm.c: mux serve emits a per-turn "PROF" protocol line next to TIERS/HITS
(window deltas per request, same convention as the STAT hit%); the phase
window base is now always captured (a few loads per request).
- openai_server.py: parses PROF into a 120-turn rolling window and serves it
at /profile (read-only, same trust level as /health).
- web: new Profiling tab — stat tiles (tok/s, wall, tokens/forward, disk
service), wall-time composition bars for the last turn and the window,
per-turn throughput and stacked phase columns with hover readouts, and a
table of recent turns. Disk service is shown apart from the stack: it
overlaps with compute, so only the I/O wait the compute thread felt counts
inside wall time. Phase colours are a CVD-validated set with gaps + legend
+ table so identity never rides on colour alone.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WhTmF8yvEBgSkUKSVfZF7P
run_serve_mux (SERVE_BATCH, used by openai_server.py / coli web) completes
requests in mux_done, so the run_serve per-turn hook never fired there.
Snapshot the window where hits0 is taken, record batched-forward latency
around step_decode_batch, report on stderr at DONE. With KV_SLOTS>1 the
window shares batched forwards across slots — same convention as the
existing STAT hit%.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TCBNCciBaHea41QmidLUMn
Vincenzo Fornaro