cuda: enable pipe_layer_sparse resident pipeline at decode (S>=1 gate)

One-line gate change: relax the pipe2 call-site gate from S>=8 to S>=1,
allowing the existing resident-pipeline (pipe_layer_sparse) to run during
single-token decode, not just prefill.

The S>=8 gate was a performance heuristic (prefill-only), not a correctness
constraint — pipe_layer_sparse is fully S-general. At decode it keeps the
residual stream x on the GPU device across all 78 layers, running rmsnorm,
residual adds, and shared-expert matmuls on-device. This eliminates the
~12.5k GPU sync interruptions per decode that caused the expert-matmul
regression (13.3s -> 9.2s), and moves the untracked 'other' CPU work
(rmsnorms, residual adds, routing) onto the GPU (29s -> 17.6s).

Measured (GLM-5.2 744B int4, RTX 5070 Ti, 32GB RAM, budget=4 + full disk stack):
  tok/s: 0.72 -> 1.07  (+49%)
  decode: 44.5s -> 29.9s  (-33%)
  expert-matmul: 13.3s -> 9.2s  (regression fixed)
  'other': 29s -> 17.6s  (-39%)

Correctness: 32/32 oracle positions (3 consecutive runs).
Configuration: COLI_CUDA=1 CUDA_DENSE=1 COLI_CUDA_ATTN=1 COLI_CUDA_PIPE=2
  CUDA_EXPERT_GB=0 EXPERT_BUDGET=4 PIPE=1 RAM_GB=28 PILOT_REAL=1 DIRECT=1
This commit is contained in:
woolcoxm
2026-07-15 12:25:16 -04:00
parent 25219de45b
commit c5b1d14a74
+1 -1
View File
@@ -3188,7 +3188,7 @@ static void layers_forward_rows(Model *m, float *x, int S, int pos_base,
* ai confini di layer. x host diventa STALE finche' la residenza e' attiva. */
float *x_dev=NULL; int x_dev_on=-1;
size_t xb=(size_t)S*(size_t)D*4;
int pipe2 = g_cuda_pipe>=2 && !kvs && S>=8 && g_cuda_enabled && c->kv_lora<=512 &&
int pipe2 = g_cuda_pipe>=2 && !kvs && S>=1 && g_cuda_enabled && c->kv_lora<=512 &&
!(m->has_dsa && pos_base+S>c->index_topk);
#endif
for(int i=0;i<c->n_layers;i++){