ec89136029
* Fuse CUDA expert MLP execution * Group CUDA expert transfers by device * Instrument grouped CUDA expert execution * Bound grouped CUDA decode scratch * Execute expert groups across GPUs in parallel * Release host backing for multi-GPU experts * Define quality-preserving memory policies * Overlap cold expert loading with resident compute * Adapt expert placement with session LFRU * Fuse q4 expert gate and up dispatch * Plan CPU work on physical cores * Batch grouped expert CUDA kernels * Separate VRAM and RAM expert placement * Add ragged multi-sequence decode forward * feat(runtime): add continuous decode scheduler * Route concurrent API requests through batch scheduler * Harden multiplex request lifecycle and framing * Cancel disconnected multiplex requests * Bind API port before starting the engine * fix automatic KV slot allocation * add native int4 Tensor Core grouped GEMM * add Tensor Core throughput benchmark * optimize packed int4 low-row kernels * add asynchronous CUDA staging streams * document validated six-GPU dense acceleration * tune six-GPU expert hot set * raise validated expert hot-set target * add CUDA MLA absorption core * fuse grouped expert gate and up projections * Warn for explicit lossy routing flags * Add full-resident expert placement mode * Adapt VRAM expert slots to live routes * Accelerate int4 matvec on AVX-512 * Reduce AVX-512 and RoPE decode overhead * Seed every GPU expert layer after prefill * Limit live GPU swaps during decode * CUDA batch MLA attention, kv_b head-sharding, fused o_proj, expert-group dispatch, W4A16 kernels Lab-qualified on the 6x RTX 5090 machine (914-token request benchmark): - batch MLA absorption kernel (COLI_CUDA_ATTN=1): whole-batch attention on device, 154.8s -> 102.4s - attention -> o_proj fusion on the layer device: -> 97.4s - kv_b head-sharding across cards (COLI_CUDA_ATTN_SHARD=1), no weight duplication: -> 94.05s - per-device expert-group dispatch with pinned-buffer async transfers, W4A16 tensor-core kernels for the shared expert, OMP hot-thread tuning Negative results (reverted, kept out): GPU-side weighted scatter-add (atomics + per-layer D2H lose 43.8%), shared-expert fused small-batch kernel (-38.8%), W4A4 grouped tensor cores (int4 activations corrupt output). Details in the lab research log. * GPU resident pipeline: device-resident prefill attention chain, GPU expert groups in prefill, batched router, W4A16 mixed dispatch COLI_CUDA_PIPE=1 keeps the prefill data plane on the layer home device; control flow (routing, cache/pin management) stays on CPU. Any CUDA failure falls back to the unchanged CPU path. - Device primitives + unit tests (tests/test_pipe_cuda.cu): rmsnorm (strided), interleaved RoPE, silu-mul, residual add, fixed-order row merge (no atomics), device-input GEMM, persistent per-device scratch. All verified against the engine's CPU math on SM120 (worst 1.2e-5). - attn_pipe_prefill: q_a -> norm -> q_b -> rope -> kv_a -> norm -> rope -> batch attention -> o_proj in one device chain (q_a/q_b/kv_a colocated with kv_b); only the final [S,D] and the new KV rows return to host. Attention 41.2s -> 30.8s on the 1571-token benchmark. - Prefill batch-union now uses the GPU expert groups (previously gated to S<=64, leaving all VRAM-resident experts idle during prefill - measured 21ms of GPU expert time in a 148s prefill). Expert phase 78.9s -> 69.0s. - Router computed as one batched matmul instead of S sequential rows (bit-identical math). - W4A16 tensor-core path for expert groups (COLI_CUDA_TC_W4A16=1) with row-count mixed dispatch: >=16 rows per expert use tensor cores, smaller batches keep the naive kernel (tensor cores measured negative below ~16 rows). Expert phase 69.0s -> 64.3s, decode unaffected. Net on the 1571-token prefill benchmark: 148.8s -> 114.3-126.8s (component timings stable across runs; wall drifts +-3-5s because .coli_usage placement learning shifts the expert tiers between runs). PROFILO now also prints the prefill-phase breakdown. * Skip OMP hot-thread tuning when CUDA is enabled The active-spin worker team measured 66.9s->20.9s on the CPU-only Zen5 build, but on the six-GPU full-residency workload the spinning workers contend with the CUDA dispatch threads: ~4x slower prefill with the process stuck near 1.8 cores. Gate the tuning on COLI_CUDA so each configuration keeps the behavior it was measured to prefer. * Inc.2a: sparse layers fully resident on the layer device, residual hops cards at layer boundaries COLI_CUDA_PIPE=2 keeps the residual stream on the layer home device for consecutive sparse layers (cudaMemcpyPeer at boundaries): in/post norms, attention chain, both residual adds and the shared-expert MLP run on device. Per layer only the post-norm activations (router + CPU-tier experts + group gather), the new KV rows and, on DSA indexer layers, the pre-attention norm leave the card. Per-layer transfers drop from ~130MB to ~70MB. A device-side snapshot at layer entry makes any mid-layer CUDA failure fall back to the unchanged CPU path idempotently. 1571-token prefill: 127.1s (PIPE=1 control) -> 117.6/118.9s, components attention 30.8->26.1, other 31.8->22.5-24.5; output verified coherent against the control. * Head-sharded attention inside the pipe: negative on PCIe star topology, gated opt-in Slicing q per card from the home device and collecting ctx back serializes ~95MB/layer through the home card's PCIe link: attention 26.1s -> 41.4/44.4s on the 1571-token benchmark (two repeats), wall 117.6 -> 135-138s. The standalone host-path sharding won because six cards uploaded from host RAM in parallel; a home-device star has no such parallelism without NVLink. Kept behind COLI_CUDA_PIPE_SHARD=1 for interconnects where peer bandwidth does not share one root port. * Inc.3: device-resident KV shadow for decode attention Decode re-uploaded the whole latent+rope window per layer per token (~300MB/token at 1571 context). Each layer now keeps a device shadow of the compressed KV on its kv_b card, bulk-synced when behind and appended incrementally; the host cache stays canonical. Invalidation on kv_bind (slot switch), kv_alloc (resize) and on any overwrite of mirrored rows, with the legacy full-upload path as fallback. Measured (COLI_CUDA_PIPE gate): short-context decode 5.48 -> 5.59/5.87 tok/s, 1571-context decode 4.14 -> 4.22 tok/s. Decode remains CPU-expert bound; the shadow removes the transfer tax, not the compute. * tools: unified user-experience benchmark (bench_ux.sh) Two fixed scenarios (short chat, long-document QA), TTFT + decode tok/s + first-line drift check, TEMP=0 DRAFT=0 enforced, medians over REPS runs. Encodes the measurement discipline from the lab record: same binary per comparison, judge medians because .coli_usage placement learning drifts wall times between runs. * tools: bench_ux.sh executable bit * gitignore compiled test binaries * tools: expert_atlas.py — measure per-expert topic affinity (#175) Diffs .coli_usage across 10 themed probe batches (code/math/chinese/ prose/science/law/poetry/structured/translation/casual, 3 prompts each) driven through a running API server — one engine load total. Every touched expert gets a topic-affinity vector, entropy, and a specialist/ generalist label; output experts.json feeds the Brain page hover. * serve: persist .coli_usage after every turn in mux mode, not only at exit run_serve_mux saved the learning cache once at shutdown; a crash lost the whole session's routing history, and live consumers of the file (expert_atlas.py diffs it between probe batches) saw a frozen snapshot. Now saved per turn like the interactive path (165KB write, negligible). * web: Brain hover shows measured expert atlas when published If /experts.json (from tools/expert_atlas.py, #175) is served next to the app, the tooltip upgrades from the depth heuristic to measured data: specialist/generalist label, entropy, and the top-3 topic affinities. Row index maps to real layer (row+3, last row = MTP 78). Falls back to the heuristic when no atlas is published. --------- Co-authored-by: JustVugg <JustVugg@users.noreply.github.com>
181 lines
8.7 KiB
TypeScript
181 lines
8.7 KiB
TypeScript
import { useEffect, useRef, useState } from "react"
|
||
import { BrainCircuit, Flame, Layers } from "lucide-react"
|
||
|
||
import { endpoint } from "@/lib/api"
|
||
|
||
interface ExpertMap { rows: number; cols: number; map: string; hits: string; seq: number }
|
||
interface AtlasEntry { affinity: Record<string, number>; entropy: number; top: string; label: string }
|
||
|
||
const TIER_NAME = ["Disk", "RAM", "VRAM"]
|
||
const TIER_RGB: [number, number, number][] = [[58, 71, 80], [90, 155, 216], [78, 214, 165]]
|
||
|
||
/* Layer-depth heuristic: what this region of the network tends to specialise in.
|
||
* Honest framing — these are the depth roles observed across MoE interpretability
|
||
* work, not per-expert ground truth (that needs co-activation analysis, #119). */
|
||
function depthRole(row: number, rows: number, isMtp: boolean): string {
|
||
if (isMtp) return "MTP head — drafts the next token for speculative decoding"
|
||
const f = row / Math.max(rows - 1, 1)
|
||
if (f < 0.2) return "early layers — surface features: tokens, spelling, local syntax"
|
||
if (f < 0.45) return "lower-middle — phrase structure, word relations, simple facts"
|
||
if (f < 0.7) return "upper-middle — semantics, long-range context, reasoning steps"
|
||
if (f < 0.9) return "late layers — planning the answer, style, coherence"
|
||
return "final layers — output shaping: picks the actual next-token distribution"
|
||
}
|
||
|
||
export function Brain({ baseUrl, apiKey, connected }: { baseUrl: string; apiKey: string; connected: boolean }) {
|
||
const canvasRef = useRef<HTMLCanvasElement>(null)
|
||
const wrapRef = useRef<HTMLDivElement>(null)
|
||
const [wrapSize, setWrapSize] = useState({ w: 1200, h: 700 })
|
||
const [data, setData] = useState<ExpertMap | null>(null)
|
||
const [atlas, setAtlas] = useState<Record<string, AtlasEntry> | null>(null)
|
||
const [tip, setTip] = useState<{ x: number; y: number; row: number; col: number; tier: number; heat: number } | null>(null)
|
||
const pulseRef = useRef<Float32Array | null>(null) // per-expert pulse intensity 0..1
|
||
const lastSeq = useRef(0)
|
||
const rafRef = useRef(0)
|
||
|
||
// load the expert atlas if published (measured topic affinity, #175)
|
||
useEffect(() => {
|
||
fetch("/experts.json").then(r => r.ok ? r.json() : null).then(d => {
|
||
if (d?.experts) setAtlas(d.experts)
|
||
}).catch(() => {})
|
||
}, [])
|
||
|
||
// track container size for responsive cell sizing
|
||
useEffect(() => {
|
||
const el = wrapRef.current
|
||
if (!el) return
|
||
const ro = new ResizeObserver(() => {
|
||
setWrapSize({ w: el.clientWidth - 24, h: el.clientHeight - 24 })
|
||
})
|
||
ro.observe(el)
|
||
return () => ro.disconnect()
|
||
}, [])
|
||
|
||
// poll /experts
|
||
useEffect(() => {
|
||
if (!connected) return
|
||
let disposed = false
|
||
const base = baseUrl.replace(/\/v1\/?$/, "")
|
||
const poll = async () => {
|
||
try {
|
||
const res = await fetch(endpoint(base, "/experts"), { headers: apiKey ? { Authorization: `Bearer ${apiKey}` } : {} })
|
||
const next = (await res.json()) as ExpertMap
|
||
if (disposed || !next.rows) return
|
||
setData(next)
|
||
if (next.seq !== lastSeq.current && next.hits) {
|
||
lastSeq.current = next.seq
|
||
const n = next.rows * next.cols
|
||
if (!pulseRef.current || pulseRef.current.length !== n) pulseRef.current = new Float32Array(n)
|
||
const p = pulseRef.current
|
||
for (let i = 0; i < n; i++) {
|
||
const byte = parseInt(next.hits.substr((i >> 3) * 2, 2), 16) || 0
|
||
if (byte & (1 << (i & 7))) p[i] = 1
|
||
}
|
||
}
|
||
} catch { /* engine busy or restarting — keep the last frame */ }
|
||
}
|
||
void poll()
|
||
const t = window.setInterval(() => void poll(), 1500)
|
||
return () => { disposed = true; window.clearInterval(t) }
|
||
}, [baseUrl, apiKey, connected])
|
||
|
||
// render loop: grid + decaying pulses
|
||
useEffect(() => {
|
||
const canvas = canvasRef.current
|
||
if (!canvas || !data) return
|
||
const ctx = canvas.getContext("2d")
|
||
if (!ctx) return
|
||
const { rows, cols, map } = data
|
||
const cell = Math.max(2, Math.floor(Math.min(wrapSize.w / cols, wrapSize.h / rows)))
|
||
const gap = cell >= 4 ? 1 : 0
|
||
canvas.width = cols * (cell + gap)
|
||
canvas.height = rows * (cell + gap)
|
||
|
||
const draw = () => {
|
||
ctx.clearRect(0, 0, canvas.width, canvas.height)
|
||
const p = pulseRef.current
|
||
for (let r = 0; r < rows; r++) {
|
||
for (let c = 0; c < cols; c++) {
|
||
const i = r * cols + c
|
||
const byte = parseInt(map.substr(i * 2, 2), 16) || 0
|
||
const tier = byte >> 6
|
||
const heat = byte & 63
|
||
const [R, G, B] = TIER_RGB[tier] ?? TIER_RGB[0]
|
||
// heat scales brightness: cold experts dim, hot experts full colour
|
||
const lum = 0.35 + 0.65 * Math.min(heat / 24, 1)
|
||
let rr = R * lum, gg = G * lum, bb = B * lum
|
||
const pulse = p ? p[i] : 0
|
||
if (pulse > 0.01) { rr += (255 - rr) * pulse; gg += (255 - gg) * pulse; bb += (255 - bb) * pulse }
|
||
ctx.fillStyle = `rgb(${rr | 0},${gg | 0},${bb | 0})`
|
||
ctx.fillRect(c * (cell + gap), r * (cell + gap), cell, cell)
|
||
}
|
||
}
|
||
let alive = false
|
||
if (p) for (let i = 0; i < p.length; i++) { if (p[i] > 0.01) { p[i] *= 0.94; alive = true } else p[i] = 0 }
|
||
if (alive) rafRef.current = requestAnimationFrame(draw)
|
||
}
|
||
draw()
|
||
const keepalive = window.setInterval(() => { if (!rafRef.current) draw(); rafRef.current = 0 }, 400)
|
||
return () => { cancelAnimationFrame(rafRef.current); window.clearInterval(keepalive) }
|
||
}, [data, wrapSize])
|
||
|
||
const onMove = (e: React.MouseEvent<HTMLCanvasElement>) => {
|
||
if (!data) return
|
||
const rect = e.currentTarget.getBoundingClientRect()
|
||
const scaleX = e.currentTarget.width / rect.width
|
||
const scaleY = e.currentTarget.height / rect.height
|
||
const cell = Math.max(2, Math.floor(Math.min(wrapSize.w / data.cols, wrapSize.h / data.rows)))
|
||
const gap = cell >= 4 ? 1 : 0
|
||
const col = Math.floor(((e.clientX - rect.left) * scaleX) / (cell + gap))
|
||
const row = Math.floor(((e.clientY - rect.top) * scaleY) / (cell + gap))
|
||
if (row < 0 || row >= data.rows || col < 0 || col >= data.cols) { setTip(null); return }
|
||
const byte = parseInt(data.map.substr((row * data.cols + col) * 2, 2), 16) || 0
|
||
setTip({ x: e.clientX, y: e.clientY, row, col, tier: byte >> 6, heat: byte & 63 })
|
||
}
|
||
|
||
const totals = data ? (() => {
|
||
const t = [0, 0, 0]
|
||
for (let i = 0; i < data.rows * data.cols; i++) t[(parseInt(data.map.substr(i * 2, 2), 16) || 0) >> 6]++
|
||
return t
|
||
})() : [0, 0, 0]
|
||
|
||
return (
|
||
<div className="brain-page">
|
||
<div className="brain-head">
|
||
<div className="section-title"><BrainCircuit className="size-4" /> Expert Cortex — {data ? `${data.rows} layers × ${data.cols} experts` : "waiting for engine"}</div>
|
||
<div className="brain-legend">
|
||
<span><i style={{ background: "#4ed6a5" }} /> VRAM {totals[2].toLocaleString()}</span>
|
||
<span><i style={{ background: "#5a9bd8" }} /> RAM {totals[1].toLocaleString()}</span>
|
||
<span><i style={{ background: "#3a4750" }} /> Disk {totals[0].toLocaleString()}</span>
|
||
<span><Flame className="size-3" /> brightness = routing heat</span>
|
||
<span className="brain-pulse-hint">⚡ white flash = routed this turn</span>
|
||
</div>
|
||
</div>
|
||
<div className="brain-canvas-wrap" ref={wrapRef}>
|
||
<canvas ref={canvasRef} onMouseMove={onMove} onMouseLeave={() => setTip(null)} />
|
||
{!connected && <p className="runtime-unavailable">Connect to the engine to see the cortex.</p>}
|
||
</div>
|
||
{tip && data && (() => {
|
||
const isMtp = tip.row === data.rows - 1
|
||
const realLayer = isMtp ? 78 : tip.row + 3
|
||
const entry = atlas?.[`${realLayer}:${tip.col}`]
|
||
return (
|
||
<div className="brain-tip" style={{ left: tip.x + 14, top: tip.y + 14 }}>
|
||
<div className="brain-tip-title"><Layers className="size-3" /> Layer {realLayer}{isMtp ? " (MTP)" : ""} · Expert {tip.col}</div>
|
||
<div>Tier: <strong style={{ color: ["#8b9aa3", "#5a9bd8", "#4ed6a5"][tip.tier] }}>{TIER_NAME[tip.tier]}</strong></div>
|
||
<div>Heat: <strong>{tip.heat === 0 ? "never routed" : `~2^${tip.heat} selections`}</strong></div>
|
||
{entry ? <>
|
||
<div className={entry.label.startsWith("specialist") ? "brain-tip-spec" : undefined}>
|
||
{entry.label.startsWith("specialist") ? `⭐ Specialist: ${entry.top}` : "Generalist"}
|
||
<small> (entropy {entry.entropy})</small>
|
||
</div>
|
||
<div className="brain-tip-aff">{Object.entries(entry.affinity).sort((a, b) => b[1] - a[1]).slice(0, 3)
|
||
.map(([c, p]) => `${c} ${Math.round(p * 100)}%`).join(" · ")}</div>
|
||
</> : <div className="brain-tip-role">{depthRole(tip.row, data.rows, isMtp)}</div>}
|
||
</div>
|
||
)
|
||
})()}
|
||
</div>
|
||
)
|
||
}
|