cuda: enable pipe_layer_sparse resident pipeline at decode (S>=1 gate)
One-line gate change: relax the pipe2 call-site gate from S>=8 to S>=1, allowing the existing resident-pipeline (pipe_layer_sparse) to run during single-token decode, not just prefill. The S>=8 gate was a performance heuristic (prefill-only), not a correctness constraint — pipe_layer_sparse is fully S-general. At decode it keeps the residual stream x on the GPU device across all 78 layers, running rmsnorm, residual adds, and shared-expert matmuls on-device. This eliminates the ~12.5k GPU sync interruptions per decode that caused the expert-matmul regression (13.3s -> 9.2s), and moves the untracked 'other' CPU work (rmsnorms, residual adds, routing) onto the GPU (29s -> 17.6s). Measured (GLM-5.2 744B int4, RTX 5070 Ti, 32GB RAM, budget=4 + full disk stack): tok/s: 0.72 -> 1.07 (+49%) decode: 44.5s -> 29.9s (-33%) expert-matmul: 13.3s -> 9.2s (regression fixed) 'other': 29s -> 17.6s (-39%) Correctness: 32/32 oracle positions (3 consecutive runs). Configuration: COLI_CUDA=1 CUDA_DENSE=1 COLI_CUDA_ATTN=1 COLI_CUDA_PIPE=2 CUDA_EXPERT_GB=0 EXPERT_BUDGET=4 PIPE=1 RAM_GB=28 PILOT_REAL=1 DIRECT=1
This commit is contained in:
@@ -3188,7 +3188,7 @@ static void layers_forward_rows(Model *m, float *x, int S, int pos_base,
|
||||
* ai confini di layer. x host diventa STALE finche' la residenza e' attiva. */
|
||||
float *x_dev=NULL; int x_dev_on=-1;
|
||||
size_t xb=(size_t)S*(size_t)D*4;
|
||||
int pipe2 = g_cuda_pipe>=2 && !kvs && S>=8 && g_cuda_enabled && c->kv_lora<=512 &&
|
||||
int pipe2 = g_cuda_pipe>=2 && !kvs && S>=1 && g_cuda_enabled && c->kv_lora<=512 &&
|
||||
!(m->has_dsa && pos_base+S>c->index_topk);
|
||||
#endif
|
||||
for(int i=0;i<c->n_layers;i++){
|
||||
|
||||
Reference in New Issue
Block a user