cuda: grouped-int4 (fmt=4) support in the expert-group kernels (#334)

The grouped MoE kernels were per-row-only: GroupDesc had no group-size
fields, row_bytes() returned 0 for fmt=4, the scale buffer was hardcoded
to O floats, and a fmt=4 group that reached the generic path would have
been silently decoded as int2. This closed the GPU expert tier to every
grouped container — including the g64 quality line (#225) and the E8
lattice route (#347) whose whole point is fitting more experts in VRAM.

- ColiCudaTensor gains gs/scale_count; upload allocates O*ceil(I/gs)
  scales for fmt=4 and applies the same offset->signed nibble conversion
  as fmt=2 (identical packing). New ABI entry coli_cuda_tensor_upload_g
  carries gs without touching the existing symbol — an old Windows DLL
  missing it returns 0 and the tensor simply stays CPU-side.
- GroupDesc gains per-tensor group sizes; new grouped_hidden_g4_dual /
  grouped_down_g4 apply the per-group scale inside the accumulation
  (gs is required even, so a packed byte never straddles groups; gs=0
  degrades to per-row, letting fmt=2 members ride the same launch).
- coli_cuda_expert_group routes any group containing fmt=4 through the
  g4 kernels; pure-fmt=2 groups keep the existing paths byte-identical.
  The generic fallback now explicitly rejects fmt=4 instead of decoding
  garbage (#334's prevention note, made real).

tests/test_grouped_g4_cuda.cu: kernel-vs-CPU oracle over 50 trials x 3
experts — gs=64, a non-divisible tail group (200 % 64), and a per-row
member in the same launch: zero mismatches on a 5090.

make check 77/77; CPU, CUDA and MinGW builds clean.
This commit is contained in:
ZacharyZcR
2026-07-20 08:56:49 +08:00
parent 61004dcb84
commit a74e3e0c3a
5 changed files with 206 additions and 8 deletions
+5
View File
@@ -235,6 +235,11 @@ static void qt_cuda_reset(QT *t){
static int qt_cuda_upload(QT *t){
const void *weights = t->fmt==0 ? (const void*)t->qf
: t->fmt==1 ? (const void*)t->q8 : (const void*)t->q4;
if(t->fmt==4) /* grouped int4 (#334): scales are [O, ceil(I/gs)] — the plain
* upload would truncate them to O floats and the group kernels
* would read garbage. An old DLL without the _g symbol returns 0
* and the tensor simply stays CPU-side. */
return coli_cuda_tensor_upload_g(&t->cuda,weights,t->s,t->fmt,t->I,t->O,t->cuda_device,t->gs);
return coli_cuda_tensor_upload(&t->cuda,weights,t->s,t->fmt,t->I,t->O,t->cuda_device);
}
static int qt_cuda_update(QT *t){