url/colibri
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge pull request #342 from ZacharyZcR/feat/expert-group-overlap

Browse Source 
cuda: COLI_GROUP_ASYNC=1 — overlap the CPU expert rows with the GPU groups at decode (opt-in, +6-8%)
This commit is contained in:
Vincenzo Fornaro
2026-07-20 18:07:02 +02:00
committed by GitHub
parent e78fcfcbca 7d01d21023
commit 06961728d8
4 changed files with 258 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
c/backend_cuda.h
+10
View File
@@ -70,6 +70,16 @@ COLI_CUDA_DLLEXPORT int coli_cuda_shared_mlp_w4a16(ColiCudaTensor *gate, ColiCud
/* Packed group of same-shaped experts. Inputs and outputs contain sum(rows)
* consecutive [D] rows in call order. */
/* Async issue/take split of the group call below (Inc.4): issue launches on the
* device stream and returns; take syncs and returns the pinned result rows (valid
* until the next issue on that device). Small totals only (<=8 rows); one
* outstanding issue per device. */
COLI_CUDA_DLLEXPORT int coli_cuda_expert_group_issue(ColiCudaTensor *const *gates,
ColiCudaTensor *const *ups,
ColiCudaTensor *const *downs,
const int *rows, int count, const float *x);
COLI_CUDA_DLLEXPORT const float *coli_cuda_expert_group_take(int device);
COLI_CUDA_DLLEXPORT int coli_cuda_expert_group(ColiCudaTensor *const *gates,
ColiCudaTensor *const *ups,
ColiCudaTensor *const *downs,