url/colibri
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

cuda: COLI_GROUP_ASYNC=1 — async expert-group issue/take with CPU/GPU overlap at decode (opt-in, +6-8% measured)

Browse Source
This commit is contained in:
ZacharyZcR
2026-07-17 15:32:09 +08:00
parent 61004dcb84
commit ab55f4900c
3 changed files with 213 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
c/backend_cuda.h
+10
View File
@@ -67,6 +67,16 @@ COLI_CUDA_DLLEXPORT int coli_cuda_shared_mlp_w4a16(ColiCudaTensor *gate, ColiCud
/* Packed group of same-shaped experts. Inputs and outputs contain sum(rows)
* consecutive [D] rows in call order. */
/* Async issue/take split of the group call below (Inc.4): issue launches on the
* device stream and returns; take syncs and returns the pinned result rows (valid
* until the next issue on that device). Small totals only (<=8 rows); one
* outstanding issue per device. */
COLI_CUDA_DLLEXPORT int coli_cuda_expert_group_issue(ColiCudaTensor *const *gates,
ColiCudaTensor *const *ups,
ColiCudaTensor *const *downs,
const int *rows, int count, const float *x);
COLI_CUDA_DLLEXPORT const float *coli_cuda_expert_group_take(int device);
COLI_CUDA_DLLEXPORT int coli_cuda_expert_group(ColiCudaTensor *const *gates,
ColiCudaTensor *const *ups,
ColiCudaTensor *const *downs,