# GLM-5.2 on 6x RTX 5090

This note records the July 12, 2026 experiments that compared colibri with
vLLM-Moet on one machine with six RTX 5090 GPUs (32 GiB each), dual Intel Xeon
Silver 4510 (24 physical cores / 48 threads), 251 GiB of RAM, and local NVMe
storage. It exists to preserve the measured facts and avoid repeating the same
dead ends.

The day had two halves. The morning runs (below) measured colibri at 0.12
tok/s and vLLM-Moet at 2.3-2.7 tok/s. The afternoon full-resident work on
colibri (see "colibri full-resident experiments") removed disk from the decode
path entirely and reached 6.28 tok/s on the fixed 96-token greedy benchmark
and 6.84 tok/s on a 256-token run — the fastest single-request result measured
on this machine so far.

## Models and artifacts

- Official NVFP4 model: `/data/models/GLM-5.2-NVFP4` (434 GB, 47 shards)
- colibri INT4 model: `/data/models/GLM-5.2-colibri-int4` (144 shards)
- vLLM-Moet TP4 pack: `/data/models/GLM-5.2-NVFP4-packs` (190 GB)
- vLLM-Moet TP2/PP3 pack: `/data/models/GLM-5.2-NVFP4-packs-tp2pp3` (190 GB)
- vLLM-Moet checkout: `/data/test/vLLM-Moet`
- Built image: `vllm-moet-sm120:v024`

The official model was downloaded through `hf-mirror.com` with
`HF_HUB_DISABLE_XET=1`. Direct Hugging Face access failed and Xet/CAS stalled;
ordinary HTTP from the mirror sustained roughly 22-46 MB/s.

## Kernel validation

All six RTX 5090 cards passed the real 2-bit and NVFP4 delta kernel smoke
tests. Representative correctness results were:

| Path | Shape | Max relative error | Cosine similarity |
| --- | --- | ---: | ---: |
| Decode | T=2 | 2.086e-2 | 0.999878 |
| NVFP4 delta | T=2 | 1.818e-2 | 0.999884 |
| AFRAG prefill | T=128 | 2.591e-2 | 0.999883 |

The kernels work on SM120. The performance limitation below is therefore not
an unsupported-GPU fallback.

## Real generation results

All decode rates below are for a single request, not aggregate throughput.

| Runtime and layout | GPU use | TTFT | Decode rate | Important observation |
| --- | --- | ---: | ---: | --- |
| colibri INT4 (morning, partial residency) | 6 GPUs | about 42 s | 0.12 tok/s | GPU hot tier was active, but dense compute remained on CPU |
| colibri INT4 (afternoon, full residency) | 6 GPUs + 24 cores | — | 6.28-6.84 tok/s | all 19,456 experts resident in VRAM+RAM, zero disk waits |
| vLLM-Moet TP4, 12 GiB expert cache/card | GPUs 0-3 | 2.03 s | median 2.5 tok/s | 25.3% coverage; 100% whole-step replay |
| vLLM-Moet TP4, 14 GiB expert cache/card | GPUs 0-3 | not remeasured | median 2.6 tok/s | 29.5% coverage; replay still 100% |
| vLLM-Moet TP2 x PP3 | all 6 GPUs | 5.39 s | 1.78 tok/s | more local coverage, but serial PP and communication made latency worse |

The TP4 runs measured 2.3-2.7 tok/s. Raising the expert cache gained only
about 0.1 tok/s because every decode step still missed experts and replayed the
whole step. TP2/PP3 raised local expert coverage to roughly 55-62%, but did not
reduce replay enough to offset pipeline and PYNCCL communication costs.

The morning colibri run generated 16 tokens in 132.46 seconds. Its measured
hot-expert tier was 77.48 GB across six GPUs, RAM peaked at 175.94 GB, and the
expert hit rate was 71%. The largest costs were expert matmul (81.37 s), disk
reads (29.31 s), and attention (14.28 s). The afternoon experiments below
showed the dominant problem was expert loading from disk, not the CUDA path
itself.

## colibri full-resident experiments (afternoon)

Test directory: `/data/test/colibri-full-resident`. Related PR:
[#80](https://github.com/JustVugg/colibri/pull/80) (on top of PR #68). All A/B
runs used the same binary flags, environment, prompt, `TEMP=0`, and token
count; only decode throughput is compared, never load or prefill time.

### The optimization ladder (each step cumulative)

| Change | Decode rate | Evidence |
| --- | ---: | --- |
| Baseline: 150 GB VRAM + 150 GB RAM fixed placement | 2.30 tok/s | 4.15 s of disk waits per 20 tokens |
| Full residency: all 19,456 experts in VRAM+RAM | 5.77 tok/s | 100% expert hit, zero disk waits |
| In-session dynamic GPU expert repinning (`REPIN=16`) | 6.00 tok/s | expert compute 6.76 s → 5.96 s |
| 24 physical cores, `OMP_PROC_BIND=spread OMP_PLACES=cores` | +39.6% vs unpinned | unpinned 3.64 → pinned 5.08 tok/s on the same prompt |
| Prefill corrects all 75 MoE layers once (454 ms, counted in TTFT) | 6.05-6.08 tok/s | GPU expert calls 36,865 → 37,285 |
| Decode swap cap 32 → 16 per repin round | 6.10-6.28 tok/s | swap cost 0.18 → 0.09 s/round |

The final configuration measured 6.28 tok/s on the fixed 96-token greedy
benchmark and 6.84 tok/s on a 256-token run (longer decode amortizes fixed
costs; always state the token count when quoting a rate).

Winning layout, found automatically:

```text
GPU experts:  9,343 / 19,456  (176.73 GB across six cards)
RAM experts: 10,113 / 19,456  (about 191.3 GB)
Disk service/wait during decode: 0 s
```

Run configuration: `COLI_CUDA=1 COLI_GPUS=0,1,2,3,4,5 CUDA_EXPERT_GB=auto
CUDA_DENSE=1 COLI_CUDA_ATTN=1 PIN_GB=all RAM_GB=auto`, with
`OMP_NUM_THREADS=24 OMP_PROC_BIND=spread OMP_PLACES=cores` (now the default
generated by `resource_plan.py`).

### Directions tested and rejected (all reverted)

- Faster (`REPIN=8`) or slower (`REPIN=32`) repinning, and 64 swaps per
  round: swap/sync overhead or stale GPU expert sets lose more than they gain.
  16 tokens between repins and 16 swaps per round is the current local optimum
  (8 swaps also regresses: fewer GPU calls outweigh the sync savings).
- A second prefill correction pass: overfits the short prompt's routing and
  disturbs the history-based layout (5.86 tok/s).
- Lazy demotion of evicted GPU experts (4.77 tok/s) and GPU→RAM D2H recovery
  (6.15 tok/s): both cost as much as re-reading from page cache.
- CPU/GPU expert overlap via extra pthreads: no stable gain (6.09-6.30
  tok/s); GPU expert groups are too small a fraction of step time, and the
  extra threads compete with the 24 CPU workers.
- OpenMP restructuring of RAM experts (row batching across experts,
  persistent teams): neutral or negative; libgomp already reuses hot teams,
  and interleaving experts in one static schedule destroys per-expert
  NUMA/cache locality.
- `numactl --interleave=all`, 2 MB transparent huge pages, 12-core runs,
  smaller VRAM reserve, per-layer-normalized static expert profiles, VNNI
  int4×int8 decode: all neutral or negative. Capacity and TLB are not the
  limit; raw history counts predict routing better than normalized ones.
- Next-layer expert prediction with GPU staging: the routing signal is real
  (70.6-78.9% recall, +7.8% GPU coverage) but PCIe staging contends with the
  expert/attention streams and loses end to end (5.39-5.44 tok/s). Worth
  revisiting only with dedicated streams that do not stall attention.

### Candidate not yet merged: AVX-512 int4 kernel

The lossless int4→float RAM expert kernel still uses 256-bit AVX2. A 512-bit
unpack+FMA version measured +15.0% throughput in a greedy A/B (5.12 → 5.89
tok/s, expert time −19.7%), and a dual-accumulator variant added a little more
(5.89 → 5.94 tok/s at 96 tokens). However, the changed accumulation order
produces small float differences and the 64-token outputs eventually diverge,
so it cannot be claimed bit-exact. It stays out of the tree until it has
numeric-error unit tests, a perplexity/token-consistency evaluation, and an
explicit decision on whether float-accumulation differences are acceptable.
Row blocking (2/4 output rows) looked good in short microbenchmarks but the
gain disappeared over 10,000 sustained iterations and in a 256-token run
(6.84 vs 6.83 tok/s) — a turbo/cache artifact, reverted.

### Where the time goes now

At 6.00 tok/s: expert compute 5.96 s (56%), attention 2.62 s (25%), other
2.09 s (19%), disk 0 s. A gprof sample of a 96-token greedy run puts 84.9% of
CPU time in `matmul_qt` and 8.8% in `rope_interleave`; a per-thread RoPE
sin/cos cache for repeated positions cut projection/RoPE by 13.2% (attention
−5.0%) with no math change. Scheduling-level CPU optimizations are exhausted —
the remaining expert time is the int4 matmul kernel itself.

## MTP speculative decoding at full residency (July 13)

Motivation: with all experts resident and disk waits at zero, the old
objection to speculation (a rejected draft wastes a disk read) no longer
applies, so the July 3 negative result deserved a retest. All runs used the
same binary and environment as the 16-swap configuration above, `TEMP=0`,
96 new tokens, two prompts (the standard "sky is blue" prompt, whose greedy
output degenerates into repetition, and a non-degenerate "water cycle"
prompt).

Phase 1 found the installed MTP head unusable: acceptance 0-4% at every
depth, exactly the known int4-head defect from issue #8 (the three
`out-mtp-*.safetensors` byte sizes matched the "unusable" fingerprint).
Speculation with the broken head costs 15-18% throughput — pure verification
tax.

Phase 2 replaced the head with the community int8 MTP shards
(`mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp`, ~10 GB; originals kept as
`.int4.bak`). The head then worked as designed — and speculation still lost
at every depth:

| Prompt | DRAFT | tok/s | tok/forward | Acceptance |
| --- | ---: | ---: | ---: | ---: |
| water cycle | 0 | 6.79 | 1.01 | — |
| water cycle | 1 | 6.45 | 1.81 | 79% |
| water cycle | 2 | 5.65 | 2.29 | 64% |
| water cycle | 3 | 6.13 | 3.10 | 69% |
| sky (degenerate) | 0 | 6.12 | 1.01 | — |
| sky (degenerate) | 1 | 5.51 | 1.75 | 73% |
| sky (degenerate) | 2 | 4.42 | 1.88 | 44% |
| sky (degenerate) | 3 | 3.86 / 3.81 | 2.13 | 38% |

The failure is structural, not a tuning problem. In a dense model a verify
batch of S tokens rereads the same weights once, so speculation amortizes
the dominant cost. In this MoE the dominant cost is per-unique-expert weight
processing, and the S positions of a verify batch route to mostly different
experts: measured per-forward expert time scaled almost linearly with S
(80 ms at S=1, 168 ms at S=2, 306 ms at S=4 on the water-cycle prompt).
There is no amortization to collect, and the MTP head's own forward (int8
experts, one extra layer per draft token) adds cost on top. Even 79%
acceptance at DRAFT=1 nets -5%.

Two useful facts survive the negative result. First, the int8 head is
genuinely good: 69-79% chained top-1 acceptance on clean text (the int8
shards stay installed; the int4 originals were defective). Second, the
conclusion inverts if routed-expert compute per extra position ever becomes
cheap — e.g. Tensor Core grouped GEMM computing an S=4 verify batch at
near-S=1 cost on GPU-resident experts. Revisit speculation only after that
lands. Until then, benchmark and serve runs on this machine should set
`DRAFT=0` explicitly: with a working MTP head present, the engine's
`DRAFT=-1` default auto-enables 3 drafts and silently costs 10-37%.

Logs: `/data/test/colibri-six-gpu-ab-mtp-*-20260712.log` (phase 1),
`/data/test/colibri-six-gpu-ab-mtp8-*-20260713.log` (phase 2).

## AVX-512 int4 kernel qualification (July 13)

The candidate kernel from section "Candidate not yet merged" was put through
the three measurement gates it owed. The working-tree build exposes it behind
a runtime switch (`I4_ACC512=0/1`, default on), so every comparison below
uses one binary.

**Numeric error (passed, better than the incumbent).** A standalone harness
compared the AVX-512 dual-accumulator kernel and the engine's sequential
scalar-f32 order against a double-precision oracle over the real expert row
shapes (I=6144 and I=2048), 70k+ random rows across activation scales from
0.02 to 30, a non-multiple-of-32 tail case, and an alternating-sign
cancellation stress. The AVX-512 kernel's error was 2-4x LOWER than the
scalar order in every regime (e.g. gate/up rows: max rel 2.6e-4 vs 1.1e-3,
RMS 5.7e-6 vs 1.9e-5); no non-finite values. Tree reduction accumulates less
rounding than sequential summation, so the "output divergence" is the new
kernel being closer to exact, not further.

**Quality (passed).** SCORE log-likelihood over 8 multi-domain passages (251
continuation tokens): total logprob -449.30 (AVX-512 on) vs -448.70 (off),
perplexity 5.99 vs 5.98 — a 0.24% difference with per-passage signs split
4/4. Statistically indistinguishable.

**Throughput (+4% mean, +7% where it applies).** Three fixed greedy prompts,
`TEMP=0 DRAFT=0`, 96 tokens, ABAB-interleaved on/off, two repeats each:

| Prompt | on | off | gain |
| --- | ---: | ---: | ---: |
| sky | 6.08 / 6.08 | 5.75 / 5.62 | +7.0% |
| water cycle | 6.51 / 6.50 | 6.53 / 6.52 | 0% |
| rivers | 5.79 / 5.82 | 5.24 / 5.57 | +7.4% |

The earlier +15% estimate came from a low 5.12 baseline and is retired. The
gain is workload-dependent for a structural reason: the kernel only
accelerates experts computed on CPU, and the water-cycle prompt's routing
happens to be served almost entirely from the GPU hot tier. This is the
clearest evidence yet that the next big lever is making GPU experts a
compute tier (Tensor Core grouped GEMM), which would also shrink every
CPU-side gain — and, by making multi-position verify batches cheap,
would re-open the MTP speculation door (69-79% acceptance is already
banked above).

Verdict: the kernel is numerically superior, quality-neutral, and worth
+4-7% on CPU-heavy routings at zero cost. Recommended for merge; the only
open item is the policy call that outputs are not bit-identical to the old
(worse) accumulation order. Logs:
`/data/test/colibri-six-gpu-ab-avx512q-*-20260713.log`,
`/data/test/score-acc{0,1}.out`, harness `/data/test/i4_numeric_test.c`.

## Why six 5090s did not match two RTX PRO 6000s

The reference vLLM-Moet result reports roughly 28-32 tok/s on two RTX PRO 6000
Blackwell cards. Each PRO 6000 has 96 GB of VRAM, while each RTX 5090 has 32
GB. Both machines may total 192 GB, but expert residency is constrained per
rank and runtime state is replicated. Six smaller memory islands are not
equivalent to two large ones.

The 5090 has strong compute and is cost-effective for concurrency, but this
workload is dominated by resident expert capacity and miss recovery. Changing
TP/PP layout cannot repair a policy that replays an entire step after any
expert miss.

## Conclusions for colibri

The core idea remains valid: VRAM, RAM, and NVMe are storage tiers with
different bandwidth and latency, so a model larger than VRAM can run. The
full-resident experiments strengthened it: on this machine the whole int4
model fits in VRAM+RAM, and once disk left the decode path colibri at
6.28-6.84 tok/s beat every vLLM-Moet layout measured here (best 2.6 tok/s),
because colibri never replays a whole step on an expert miss.

The remaining bottleneck is no longer storage or scheduling — it is the int4
expert matmul on CPU (56% of step time, 84.9% of CPU samples in `matmul_qt`),
with attention (25%) next in line. Priorities:

1. Qualify the AVX-512 int4 kernel (numeric-error tests, perplexity
   evaluation, fixed greedy repeat runs) and take its ~15% if it passes.
2. Add finer counters to CPU expert execution (experts per layer, rows per
   expert, per-matrix time) to guide kernel work.
3. Keep all A/B tests on fixed `TEMP=0` prompts and fixed token counts; never
   mix sampled runs into comparisons.
4. Once the expert path is stably above about 7 tok/s, optimize attention
   projection/RoPE and score-softmax-value.

Twenty to thirty tokens per second has still not been demonstrated on this
machine; 6.84 tok/s (256-token greedy) is the current ceiling.

## Last known lab state

The fastest configuration overall is colibri full-resident in
`/data/test/colibri-full-resident` (6.28-6.84 tok/s; stop the
`glm52-moet-real` service before benchmarking it and restore the service
afterwards). The fastest vLLM-Moet configuration was the TP4 service on port
8000, using GPUs 0-3 and the 14 GiB-per-card expert cache; GPUs 4-5 were idle.
Preserve both pack directories until the PP experiment no longer needs to be
reproduced. Raw A/B logs live under `/data/test/colibri-six-gpu-ab-*.log`.
These paths describe the lab machine and are not repository dependencies.
