Files
colibri/web/src/index.css
T
ZacharyZcR ec89136029 GPU resident pipeline: batch CUDA attention, head-sharded kv_b, prefill expert groups, W4A16 mixed dispatch (#111)
* Fuse CUDA expert MLP execution

* Group CUDA expert transfers by device

* Instrument grouped CUDA expert execution

* Bound grouped CUDA decode scratch

* Execute expert groups across GPUs in parallel

* Release host backing for multi-GPU experts

* Define quality-preserving memory policies

* Overlap cold expert loading with resident compute

* Adapt expert placement with session LFRU

* Fuse q4 expert gate and up dispatch

* Plan CPU work on physical cores

* Batch grouped expert CUDA kernels

* Separate VRAM and RAM expert placement

* Add ragged multi-sequence decode forward

* feat(runtime): add continuous decode scheduler

* Route concurrent API requests through batch scheduler

* Harden multiplex request lifecycle and framing

* Cancel disconnected multiplex requests

* Bind API port before starting the engine

* fix automatic KV slot allocation

* add native int4 Tensor Core grouped GEMM

* add Tensor Core throughput benchmark

* optimize packed int4 low-row kernels

* add asynchronous CUDA staging streams

* document validated six-GPU dense acceleration

* tune six-GPU expert hot set

* raise validated expert hot-set target

* add CUDA MLA absorption core

* fuse grouped expert gate and up projections

* Warn for explicit lossy routing flags

* Add full-resident expert placement mode

* Adapt VRAM expert slots to live routes

* Accelerate int4 matvec on AVX-512

* Reduce AVX-512 and RoPE decode overhead

* Seed every GPU expert layer after prefill

* Limit live GPU swaps during decode

* CUDA batch MLA attention, kv_b head-sharding, fused o_proj, expert-group dispatch, W4A16 kernels

Lab-qualified on the 6x RTX 5090 machine (914-token request benchmark):
- batch MLA absorption kernel (COLI_CUDA_ATTN=1): whole-batch attention on
  device, 154.8s -> 102.4s
- attention -> o_proj fusion on the layer device: -> 97.4s
- kv_b head-sharding across cards (COLI_CUDA_ATTN_SHARD=1), no weight
  duplication: -> 94.05s
- per-device expert-group dispatch with pinned-buffer async transfers,
  W4A16 tensor-core kernels for the shared expert, OMP hot-thread tuning

Negative results (reverted, kept out): GPU-side weighted scatter-add
(atomics + per-layer D2H lose 43.8%), shared-expert fused small-batch
kernel (-38.8%), W4A4 grouped tensor cores (int4 activations corrupt
output). Details in the lab research log.

* GPU resident pipeline: device-resident prefill attention chain, GPU expert groups in prefill, batched router, W4A16 mixed dispatch

COLI_CUDA_PIPE=1 keeps the prefill data plane on the layer home device;
control flow (routing, cache/pin management) stays on CPU. Any CUDA
failure falls back to the unchanged CPU path.

- Device primitives + unit tests (tests/test_pipe_cuda.cu): rmsnorm
  (strided), interleaved RoPE, silu-mul, residual add, fixed-order row
  merge (no atomics), device-input GEMM, persistent per-device scratch.
  All verified against the engine's CPU math on SM120 (worst 1.2e-5).
- attn_pipe_prefill: q_a -> norm -> q_b -> rope -> kv_a -> norm -> rope ->
  batch attention -> o_proj in one device chain (q_a/q_b/kv_a colocated
  with kv_b); only the final [S,D] and the new KV rows return to host.
  Attention 41.2s -> 30.8s on the 1571-token benchmark.
- Prefill batch-union now uses the GPU expert groups (previously gated to
  S<=64, leaving all VRAM-resident experts idle during prefill - measured
  21ms of GPU expert time in a 148s prefill). Expert phase 78.9s -> 69.0s.
- Router computed as one batched matmul instead of S sequential rows
  (bit-identical math).
- W4A16 tensor-core path for expert groups (COLI_CUDA_TC_W4A16=1) with
  row-count mixed dispatch: >=16 rows per expert use tensor cores, smaller
  batches keep the naive kernel (tensor cores measured negative below
  ~16 rows). Expert phase 69.0s -> 64.3s, decode unaffected.

Net on the 1571-token prefill benchmark: 148.8s -> 114.3-126.8s
(component timings stable across runs; wall drifts +-3-5s because
.coli_usage placement learning shifts the expert tiers between runs).
PROFILO now also prints the prefill-phase breakdown.

* Skip OMP hot-thread tuning when CUDA is enabled

The active-spin worker team measured 66.9s->20.9s on the CPU-only Zen5
build, but on the six-GPU full-residency workload the spinning workers
contend with the CUDA dispatch threads: ~4x slower prefill with the
process stuck near 1.8 cores. Gate the tuning on COLI_CUDA so each
configuration keeps the behavior it was measured to prefer.

* Inc.2a: sparse layers fully resident on the layer device, residual hops cards at layer boundaries

COLI_CUDA_PIPE=2 keeps the residual stream on the layer home device for
consecutive sparse layers (cudaMemcpyPeer at boundaries): in/post norms,
attention chain, both residual adds and the shared-expert MLP run on
device. Per layer only the post-norm activations (router + CPU-tier
experts + group gather), the new KV rows and, on DSA indexer layers, the
pre-attention norm leave the card. Per-layer transfers drop from ~130MB
to ~70MB. A device-side snapshot at layer entry makes any mid-layer CUDA
failure fall back to the unchanged CPU path idempotently.

1571-token prefill: 127.1s (PIPE=1 control) -> 117.6/118.9s, components
attention 30.8->26.1, other 31.8->22.5-24.5; output verified coherent
against the control.

* Head-sharded attention inside the pipe: negative on PCIe star topology, gated opt-in

Slicing q per card from the home device and collecting ctx back
serializes ~95MB/layer through the home card's PCIe link: attention
26.1s -> 41.4/44.4s on the 1571-token benchmark (two repeats), wall
117.6 -> 135-138s. The standalone host-path sharding won because six
cards uploaded from host RAM in parallel; a home-device star has no
such parallelism without NVLink. Kept behind COLI_CUDA_PIPE_SHARD=1
for interconnects where peer bandwidth does not share one root port.

* Inc.3: device-resident KV shadow for decode attention

Decode re-uploaded the whole latent+rope window per layer per token
(~300MB/token at 1571 context). Each layer now keeps a device shadow of
the compressed KV on its kv_b card, bulk-synced when behind and appended
incrementally; the host cache stays canonical. Invalidation on kv_bind
(slot switch), kv_alloc (resize) and on any overwrite of mirrored rows,
with the legacy full-upload path as fallback.

Measured (COLI_CUDA_PIPE gate): short-context decode 5.48 -> 5.59/5.87
tok/s, 1571-context decode 4.14 -> 4.22 tok/s. Decode remains CPU-expert
bound; the shadow removes the transfer tax, not the compute.

* tools: unified user-experience benchmark (bench_ux.sh)

Two fixed scenarios (short chat, long-document QA), TTFT + decode tok/s
+ first-line drift check, TEMP=0 DRAFT=0 enforced, medians over REPS
runs. Encodes the measurement discipline from the lab record: same
binary per comparison, judge medians because .coli_usage placement
learning drifts wall times between runs.

* tools: bench_ux.sh executable bit

* gitignore compiled test binaries

* tools: expert_atlas.py — measure per-expert topic affinity (#175)

Diffs .coli_usage across 10 themed probe batches (code/math/chinese/
prose/science/law/poetry/structured/translation/casual, 3 prompts each)
driven through a running API server — one engine load total. Every
touched expert gets a topic-affinity vector, entropy, and a specialist/
generalist label; output experts.json feeds the Brain page hover.

* serve: persist .coli_usage after every turn in mux mode, not only at exit

run_serve_mux saved the learning cache once at shutdown; a crash lost
the whole session's routing history, and live consumers of the file
(expert_atlas.py diffs it between probe batches) saw a frozen snapshot.
Now saved per turn like the interactive path (165KB write, negligible).

* web: Brain hover shows measured expert atlas when published

If /experts.json (from tools/expert_atlas.py, #175) is served next to
the app, the tooltip upgrades from the depth heuristic to measured
data: specialist/generalist label, entropy, and the top-3 topic
affinities. Row index maps to real layer (row+3, last row = MTP 78).
Falls back to the heuristic when no atlas is published.

---------

Co-authored-by: JustVugg <JustVugg@users.noreply.github.com>
2026-07-14 18:18:05 +02:00

154 lines
15 KiB
CSS

@import "tailwindcss";
@theme inline {
--color-background: var(--background);
--color-foreground: var(--foreground);
--color-card: var(--card);
--color-card-foreground: var(--foreground);
--color-primary: var(--primary);
--color-primary-foreground: var(--primary-foreground);
--color-secondary: var(--secondary);
--color-secondary-foreground: var(--foreground);
--color-muted-foreground: var(--muted-foreground);
--color-border: var(--border);
--color-input: var(--input);
--color-ring: var(--primary);
--color-destructive: var(--destructive);
}
:root {
color-scheme: dark;
font-family: Inter, ui-sans-serif, system-ui, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif;
--background: #080b0d;
--foreground: #e9eff0;
--card: #0d1215;
--primary: #4ed6a5;
--primary-foreground: #052118;
--secondary: #151c20;
--muted-foreground: #96a4a9;
--border: #202a2f;
--input: #10171a;
--destructive: #ff766f;
}
* { box-sizing: border-box; }
html, body, #root { min-height: 100%; margin: 0; }
body { background: var(--background); color: var(--foreground); }
button, input, textarea, select { font: inherit; }
button { cursor: pointer; }
button:focus-visible, input:focus-visible, textarea:focus-visible, select:focus-visible { outline: 2px solid var(--primary); outline-offset: 2px; }
.app-shell { min-height: 100vh; display: grid; grid-template-columns: 292px minmax(0, 1fr); background: radial-gradient(circle at 72% -20%, rgba(78,214,165,.08), transparent 34%), var(--background); }
.sidebar { position: sticky; top: 0; height: 100vh; overflow-y: auto; display: flex; flex-direction: column; gap: 24px; padding: 24px 20px; border-right: 1px solid var(--border); background: rgba(8,11,13,.86); backdrop-filter: blur(18px); }
.brand-row { display: flex; align-items: center; gap: 12px; }
.brand-mark { display: grid; place-items: center; width: 38px; height: 38px; border: 1px solid rgba(78,214,165,.3); border-radius: 12px; color: var(--primary); background: rgba(78,214,165,.08); transform: rotate(-4deg); }
.brand-row h1 { margin: 0; font-family: Georgia, serif; font-size: 22px; font-style: italic; letter-spacing: -.02em; }
.brand-row p { margin: 2px 0 0; color: var(--muted-foreground); font-size: 10px; letter-spacing: .08em; text-transform: uppercase; }
.side-section { display: grid; gap: 13px; }
.section-title { display: flex; align-items: center; gap: 7px; color: #66747a; font-size: 10px; font-weight: 700; letter-spacing: .14em; text-transform: uppercase; }
.side-section label { display: grid; gap: 7px; color: #a9b4b8; font-size: 11px; font-weight: 600; }
.side-section select { width: 100%; height: 40px; padding: 0 32px 0 11px; border: 1px solid var(--border); border-radius: 8px; color: var(--foreground); background: var(--input); outline: none; }
.field-icon { position: absolute; top: 12px; left: 12px; width: 15px; color: #647177; }
.label-line { display: flex; justify-content: space-between; align-items: center; }
.label-line code { color: var(--primary); font-size: 11px; }
.range { width: 100%; accent-color: var(--primary); }
.connection-state { display: flex; align-items: center; gap: 7px; color: #69767b; font-size: 10px; }
.connection-state span { width: 6px; height: 6px; border-radius: 99px; background: #4b5559; }
.connection-state.connected { color: #8abfa9; }.connection-state.connected span { background: var(--primary); box-shadow: 0 0 10px var(--primary); }
.runtime-section { gap: 10px; }
.runtime-grid { display: grid; grid-template-columns: 1fr 1fr; overflow: hidden; border: 1px solid var(--border); border-radius: 10px; background: var(--card); }
.runtime-grid div { display: grid; gap: 4px; padding: 10px; border-right: 1px solid var(--border); border-bottom: 1px solid var(--border); }.runtime-grid div:nth-child(2n) { border-right: 0; }.runtime-grid div:nth-last-child(-n+2) { border-bottom: 0; }
.runtime-grid span { color: #7f8d92; font-size: 9px; font-weight: 700; letter-spacing: .08em; text-transform: uppercase; }.runtime-grid strong { font: 600 16px ui-monospace, SFMono-Regular, monospace; }.runtime-grid small { color: var(--muted-foreground); font-size: 10px; font-weight: 500; }
.runtime-foot { display: flex; align-items: center; gap: 6px; color: #839197; font-size: 10px; }.runtime-foot code { margin-left: auto; color: var(--primary); }.runtime-dot { width: 6px; height: 6px; border-radius: 50%; background: var(--primary); box-shadow: 0 0 9px rgba(78,214,165,.65); }
.runtime-unavailable { margin: 0; color: var(--muted-foreground); font-size: 11px; line-height: 1.5; }.field-help { color: #718086; font-size: 9px; font-weight: 400; line-height: 1.4; }
.toggle-row { display: flex; align-items: center; justify-content: space-between; height: 42px; padding: 0 11px; border: 1px solid var(--border); border-radius: 9px; color: #a9b4b8; background: var(--input); }
.toggle-row > span { display: flex; align-items: center; gap: 8px; font-size: 11px; font-weight: 600; }.toggle-row.active { border-color: rgba(78,214,165,.35); color: var(--foreground); }
.toggle-row i { width: 30px; height: 17px; padding: 2px; border-radius: 20px; background: #293136; transition: .2s; }.toggle-row i b { display: block; width: 13px; height: 13px; border-radius: 50%; background: #78858a; transition: .2s; }.toggle-row.active i { background: rgba(78,214,165,.28); }.toggle-row.active i b { transform: translateX(13px); background: var(--primary); }
.sidebar-foot { margin-top: auto; display: flex; align-items: center; gap: 7px; color: #59666b; font-size: 10px; }
.chat-panel { min-width: 0; height: 100vh; display: grid; grid-template-rows: 72px minmax(0, 1fr) auto; }
.topbar { display: flex; align-items: center; justify-content: space-between; padding: 0 32px; border-bottom: 1px solid var(--border); }
.topbar > div:first-child { min-width: 0; display: grid; gap: 3px; }.topbar strong { overflow: hidden; font: 500 14px ui-monospace, SFMono-Regular, monospace; text-overflow: ellipsis; white-space: nowrap; }.eyebrow { color: #718086; font-size: 9px; font-weight: 800; letter-spacing: .16em; }
.top-actions { display: flex; align-items: center; gap: 8px; }.top-actions [data-slot=badge] { gap: 5px; }
.conversation { overflow-y: auto; }
.empty-state { width: min(680px, calc(100% - 40px)); min-height: 100%; margin: 0 auto; display: flex; flex-direction: column; align-items: center; justify-content: center; padding: 56px 0; text-align: center; }
.orb { position: relative; display: grid; place-items: center; width: 66px; height: 66px; margin-bottom: 24px; border: 1px solid rgba(78,214,165,.25); border-radius: 50%; color: var(--primary); background: radial-gradient(circle, rgba(78,214,165,.17), rgba(78,214,165,.02) 68%); box-shadow: 0 0 55px rgba(78,214,165,.1); }.orb svg { width: 25px; }.orb::after { content: ""; position: absolute; inset: -7px; border: 1px dashed rgba(78,214,165,.14); border-radius: inherit; }
.empty-state h2 { margin: 14px 0 14px; font-family: Georgia, serif; font-size: clamp(34px, 5vw, 55px); font-weight: 400; line-height: 1.02; letter-spacing: -.045em; }.empty-state h2 em { color: var(--primary); font-weight: 400; }.empty-state > p { max-width: 510px; margin: 0; color: var(--muted-foreground); font-size: 13px; line-height: 1.7; }
.suggestions { width: 100%; display: grid; grid-template-columns: repeat(3, 1fr); gap: 9px; margin-top: 38px; }.suggestions button { display: flex; justify-content: space-between; align-items: center; min-height: 62px; padding: 12px 14px; border: 1px solid var(--border); border-radius: 11px; color: #a5b0b4; background: rgba(13,18,21,.75); text-align: left; font-size: 11px; line-height: 1.4; transition: .2s; }.suggestions button:hover { border-color: rgba(78,214,165,.35); color: var(--foreground); transform: translateY(-2px); }
.message-list { width: min(820px, calc(100% - 48px)); margin: 0 auto; padding: 44px 0 20px; }
.message { display: grid; grid-template-columns: 34px minmax(0,1fr); gap: 14px; margin-bottom: 34px; }.avatar { display: grid; place-items: center; width: 31px; height: 31px; border: 1px solid var(--border); border-radius: 9px; color: #aeb8bc; background: var(--card); font-size: 11px; font-weight: 700; }.message.assistant .avatar { color: var(--primary); border-color: rgba(78,214,165,.25); background: rgba(78,214,165,.06); }.message-meta { height: 27px; color: #7f8c91; font-size: 10px; font-weight: 700; letter-spacing: .07em; text-transform: uppercase; }.message-body { color: #d8e0e2; font-size: 14px; line-height: 1.75; white-space: pre-wrap; overflow-wrap: anywhere; }.message.user .message-body { color: #aeb9bd; }
.typing { display: inline-flex; gap: 4px; padding-top: 6px; }.typing i { width: 5px; height: 5px; border-radius: 50%; background: var(--primary); animation: pulse 1.1s infinite; }.typing i:nth-child(2) { animation-delay: .15s; }.typing i:nth-child(3) { animation-delay: .3s; }
.composer-wrap { padding: 10px 32px 24px; background: linear-gradient(transparent, var(--background) 24%); }.composer { width: min(820px, 100%); margin: 0 auto; overflow: hidden; border: 1px solid #2a353a; border-radius: 15px; background: rgba(13,18,21,.94); box-shadow: 0 18px 55px rgba(0,0,0,.24); }.composer textarea { min-height: 74px; border: 0; }.composer-foot { display: flex; align-items: center; justify-content: space-between; padding: 0 9px 9px 15px; }.composer-foot > span { display: flex; align-items: center; gap: 6px; color: #56646a; font-size: 9px; }.error-banner { width: min(820px, 100%); margin: 0 auto 8px; padding: 9px 12px; border: 1px solid rgba(255,118,111,.25); border-radius: 9px; color: #ff9d98; background: rgba(255,118,111,.07); font-size: 11px; }
@keyframes pulse { 0%, 70%, 100% { opacity: .25; transform: translateY(0); } 35% { opacity: 1; transform: translateY(-3px); } }
@media (max-width: 820px) {
.app-shell { grid-template-columns: 1fr; }.sidebar { position: static; width: 100%; height: auto; display: grid; grid-template-columns: 1fr 1fr; border-right: 0; border-bottom: 1px solid var(--border); }.brand-row, .sidebar-foot { grid-column: 1/-1; }.chat-panel { height: 100svh; }.suggestions { grid-template-columns: 1fr; }.topbar, .composer-wrap { padding-left: 16px; padding-right: 16px; }
}
@media (max-width: 560px) { .sidebar { grid-template-columns: 1fr; }.brand-row, .sidebar-foot { grid-column: auto; }.empty-state h2 { font-size: 36px; }.message-list { width: calc(100% - 28px); }.composer-foot > span { display: none; } }
@media (prefers-reduced-motion: reduce) { *, *::before, *::after { scroll-behavior: auto !important; animation-duration: .01ms !important; animation-iteration-count: 1 !important; transition-duration: .01ms !important; } }
/* expert tier pyramid: where the 19k experts live right now */
.tier-panel { display: grid; gap: 7px; }
.tier-bar { display: flex; height: 10px; border-radius: 6px; overflow: hidden; border: 1px solid var(--border); background: var(--card); }
.tier-bar span { display: block; min-width: 0; transition: width .6s ease; }
.tier-vram { background: var(--primary); }
.tier-ram { background: #5a9bd8; }
.tier-disk { background: #3a4750; }
.tier-legend { display: flex; gap: 12px; font-size: 10px; color: #839197; flex-wrap: wrap; }
.tier-legend span { display: flex; align-items: center; gap: 5px; }
.tier-legend i { width: 8px; height: 8px; border-radius: 2px; display: inline-block; }
.tier-legend strong { font: 600 12px ui-monospace, SFMono-Regular, monospace; color: var(--foreground); }
.tier-legend small { color: var(--muted-foreground); }
/* ---- rich metrics badges & animations ---- */
.badge-live { background: rgba(78,214,165,.15); border-color: rgba(78,214,165,.4); color: #4ed6a5; }
.badge-speed { background: rgba(90,155,216,.12); border-color: rgba(90,155,216,.35); color: #5a9bd8; }
.flash { animation: flash 0.6s ease infinite alternate; }
@keyframes flash { from { opacity: 1; } to { opacity: 0.3; } }
.top-actions { display: flex; flex-wrap: wrap; gap: 6px; align-items: center; }
.top-actions .badge-live, .top-actions .badge-speed { font-variant-numeric: tabular-nums; }
/* session stats row */
.session-stats { display: flex; align-items: center; gap: 6px; font-size: 10px; color: #7f8d92; padding: 2px 0; }
.session-stats strong { font: 600 11px ui-monospace, SFMono-Regular, monospace; color: var(--foreground); }
/* tier bar tweaks */
.tier-bar { height: 12px; }
.tier-bar span { transition: width 0.8s cubic-bezier(.4,0,.2,1); }
.tier-legend strong { font-size: 13px; }
/* hardware info panel */
.hw-panel { display: grid; gap: 5px; padding: 8px 0; border-bottom: 1px solid var(--border); margin-bottom: 8px; }
.hw-row { display: flex; align-items: center; gap: 8px; font-size: 11px; color: var(--foreground); }
.hw-row svg { color: var(--primary); flex-shrink: 0; }
.hw-row small { color: var(--muted-foreground); margin-left: 4px; }
/* ---- Brain page ---- */
.view-tabs { display: flex; gap: 4px; background: var(--card); border: 1px solid var(--border); border-radius: 10px; padding: 3px; }
.view-tabs button { display: flex; align-items: center; gap: 6px; padding: 5px 14px; font-size: 12px; font-weight: 600; color: var(--muted-foreground); background: none; border: none; border-radius: 7px; cursor: pointer; }
.view-tabs button.active { background: var(--primary); color: #08110d; }
.brain-page { flex: 1; display: flex; flex-direction: column; gap: 12px; padding: 18px 22px; overflow: auto; }
.brain-head { display: flex; flex-wrap: wrap; align-items: center; justify-content: space-between; gap: 10px; }
.brain-legend { display: flex; flex-wrap: wrap; gap: 14px; font-size: 11px; color: #839197; align-items: center; }
.brain-legend i { width: 9px; height: 9px; border-radius: 2px; display: inline-block; margin-right: 5px; vertical-align: -1px; }
.brain-pulse-hint { color: var(--primary); }
.brain-canvas-wrap { flex: 1; overflow: auto; border: 1px solid var(--border); border-radius: 12px; background: #07090a; padding: 10px; }
.brain-canvas-wrap canvas { image-rendering: pixelated; max-width: 100%; cursor: crosshair; }
.brain-tip { position: fixed; z-index: 50; background: #0d1214; border: 1px solid var(--border); border-radius: 10px; padding: 10px 12px; font-size: 11px; color: var(--foreground); pointer-events: none; max-width: 280px; box-shadow: 0 8px 24px rgba(0,0,0,.5); display: grid; gap: 4px; }
.brain-tip-title { display: flex; align-items: center; gap: 5px; font-weight: 700; color: var(--primary); }
.brain-tip-role { color: #8b9aa3; font-style: italic; line-height: 1.4; border-top: 1px solid var(--border); padding-top: 5px; margin-top: 2px; }
/* Brain responsive */
.brain-canvas-wrap { display: flex; align-items: flex-start; justify-content: center; min-height: 300px; }
@media (max-width: 900px) {
.brain-page { padding: 10px; }
.brain-head { flex-direction: column; align-items: flex-start; }
.brain-legend { gap: 8px; font-size: 10px; }
.brain-tip { max-width: 220px; font-size: 10px; }
}
/* atlas hover extras */
.brain-tip-spec { color: var(--primary); font-weight: 700; }
.brain-tip-spec small, .brain-tip-aff { color: #8b9aa3; font-weight: 400; }
.brain-tip-aff { font-size: 10px; }