From c5b1d14a74f09bf9471b10d6f0680c0600227c03 Mon Sep 17 00:00:00 2001 From: woolcoxm <13604288+woolcoxm@users.noreply.github.com> Date: Wed, 15 Jul 2026 12:25:16 -0400 Subject: [PATCH] cuda: enable pipe_layer_sparse resident pipeline at decode (S>=1 gate) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit One-line gate change: relax the pipe2 call-site gate from S>=8 to S>=1, allowing the existing resident-pipeline (pipe_layer_sparse) to run during single-token decode, not just prefill. The S>=8 gate was a performance heuristic (prefill-only), not a correctness constraint — pipe_layer_sparse is fully S-general. At decode it keeps the residual stream x on the GPU device across all 78 layers, running rmsnorm, residual adds, and shared-expert matmuls on-device. This eliminates the ~12.5k GPU sync interruptions per decode that caused the expert-matmul regression (13.3s -> 9.2s), and moves the untracked 'other' CPU work (rmsnorms, residual adds, routing) onto the GPU (29s -> 17.6s). Measured (GLM-5.2 744B int4, RTX 5070 Ti, 32GB RAM, budget=4 + full disk stack): tok/s: 0.72 -> 1.07 (+49%) decode: 44.5s -> 29.9s (-33%) expert-matmul: 13.3s -> 9.2s (regression fixed) 'other': 29s -> 17.6s (-39%) Correctness: 32/32 oracle positions (3 consecutive runs). Configuration: COLI_CUDA=1 CUDA_DENSE=1 COLI_CUDA_ATTN=1 COLI_CUDA_PIPE=2 CUDA_EXPERT_GB=0 EXPERT_BUDGET=4 PIPE=1 RAM_GB=28 PILOT_REAL=1 DIRECT=1 --- c/glm.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/c/glm.c b/c/glm.c index 4f33b0c..d806301 100644 --- a/c/glm.c +++ b/c/glm.c @@ -3188,7 +3188,7 @@ static void layers_forward_rows(Model *m, float *x, int S, int pos_base, * ai confini di layer. x host diventa STALE finche' la residenza e' attiva. */ float *x_dev=NULL; int x_dev_on=-1; size_t xb=(size_t)S*(size_t)D*4; - int pipe2 = g_cuda_pipe>=2 && !kvs && S>=8 && g_cuda_enabled && c->kv_lora<=512 && + int pipe2 = g_cuda_pipe>=2 && !kvs && S>=1 && g_cuda_enabled && c->kv_lora<=512 && !(m->has_dsa && pos_base+S>c->index_topk); #endif for(int i=0;in_layers;i++){