placement: full-resident experts on large-memory hosts — CUDA_EXPERT_GB=auto, PIN_GB=all, adaptive GPU slots, RoPE cache (#80)

* Fuse CUDA expert MLP execution * Group CUDA expert transfers by device * Instrument grouped CUDA expert execution * Bound grouped CUDA decode scratch * Execute expert groups across GPUs in parallel * Release host backing for multi-GPU experts * Define quality-preserving memory policies * Overlap cold expert loading with resident compute * Adapt expert placement with session LFRU * Fuse q4 expert gate and up dispatch * Plan CPU work on physical cores * Batch grouped expert CUDA kernels * Separate VRAM and RAM expert placement * Add ragged multi-sequence decode forward * feat(runtime): add continuous decode scheduler * Route concurrent API requests through batch scheduler * Harden multiplex request lifecycle and framing * Cancel disconnected multiplex requests * Bind API port before starting the engine * fix automatic KV slot allocation * add native int4 Tensor Core grouped GEMM * add Tensor Core throughput benchmark * optimize packed int4 low-row kernels * add asynchronous CUDA staging streams * document validated six-GPU dense acceleration * tune six-GPU expert hot set * raise validated expert hot-set target * add CUDA MLA absorption core * fuse grouped expert gate and up projections * Warn for explicit lossy routing flags * Add full-resident expert placement mode * Adapt VRAM expert slots to live routes * Accelerate int4 matvec on AVX-512 * Reduce AVX-512 and RoPE decode overhead * Seed every GPU expert layer after prefill * Limit live GPU swaps during decode --------- Co-authored-by: JustVugg <JustVugg@users.noreply.github.com>
2026-07-14 22:35:39 +08:00
parent 504fcdd930
commit f1fa5bf3c8
6 changed files with 187 additions and 28 deletions
@@ -388,6 +388,23 @@ extern "C" int coli_cuda_tensor_upload(ColiCudaTensor **tensor,
    return 1;
 }

+extern "C" int coli_cuda_tensor_update(ColiCudaTensor *tensor,
+                                          const void *weights,
+                                          const float *scales) {
+    if (!tensor || !weights || (tensor->fmt && !scales)) return 0;
+    DeviceContext *ctx=find_ctx(tensor->device);
+    if (!select_ctx(ctx)) return 0;
+    if (!cuda_ok(cudaMemcpy(tensor->weights,weights,tensor->weight_bytes,
+                            cudaMemcpyHostToDevice),"tensor refresh")) return 0;
+    if(tensor->fmt==2){
+        offset_to_signed_s4<<<(unsigned)((tensor->weight_bytes+255)/256),256>>>(
+            (uint8_t*)tensor->weights,tensor->weight_bytes);
+        if(!cuda_ok(cudaGetLastError(),"int4 weight refresh")) return 0;
+    }
+    return !tensor->fmt || cuda_ok(cudaMemcpy(tensor->scales,scales,
+        (size_t)tensor->O*sizeof(float),cudaMemcpyHostToDevice),"scale refresh");
+}
+
 extern "C" int coli_cuda_matmul(ColiCudaTensor **tensor,
                                 float *y, const float *x,
                                 const void *weights, const float *scales,