url/colibri
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

web dashboard: live expert-tier panel (VRAM/RAM/disk), one-process hosting, coli web (#126)

Browse Source 
* Web dashboard: live expert-tier panel (VRAM/RAM/disk), static hosting, coli web command

- Engine: TIERS protocol line (counts + GB for VRAM/RAM/disk experts)
  emitted at READY and refreshed after every served turn, computed from
  the live pin/LRU/CUDA state.
- Server: dispatcher tracks the latest snapshot, /health exposes it as
  "tiers"; the built web UI (web/dist) is served from the same port
  (read-only, traversal-safe, SPA fallback) so the dashboard needs no
  second process.
- Web: tier bar + legend in the Runtime panel showing where the 19k
  experts live right now, with GB per tier; updates with the existing
  health polling.
- CLI: `coli web` = serve + auto-open the browser once /health answers
  (the 744B load takes minutes; a poller waits for it), --no-browser to
  disable, friendly hint when web/dist isn't built.

* coli web: HERE is a path string, not a Path

* web: same-origin default + auto-connect when served by the engine

The page hosted by coli web talked to http://127.0.0.1:8000/v1 by
default while being loaded from http://localhost:8000 - a different
origin, so the very first probe died on CORS ('Failed to fetch').
When the page is engine-served the default (and a stored stale factory
default) now resolve to window.location.origin + /v1, and the console
auto-probes on load; the Vite dev server keeps the classic default.
The server's own port is also whitelisted in DEFAULT_CORS_ORIGINS for
the localhost/127.0.0.1 cross-name case.

* web: fix auto-connect effect returning Promise (React 19 StrictMode crash)

The async connect() call inside useEffect returned a Promise that React
19 StrictMode stored as the effect cleanup. On re-render, React called
destroy_() on that Promise — 'not a function'. Block body with explicit
return undefined prevents any non-function from leaking into the cleanup
slot.

* web: rich streaming metrics — live token counter, tok/s gauge, TTFT, session totals

During generation: a flashing Zap badge counts tokens in real time.
After: Gauge shows tok/s, Timer shows TTFT (time to first token),
Layers shows prompt→completion token counts, Clock shows queue wait.
Session totals (cumulative prompt + completion) appear in the Runtime
panel. All with lucide icons and tabular-nums for stable layout.

Tier bar gains a smoother cubic-bezier transition and slightly taller
height for visual weight.

* web: hardware environment panel — CPU model, GPU count/VRAM, RAM total/free, cores

Engine emits HWINFO protocol line at startup (CPU name from /proc/cpuinfo,
core count, RAM total/available from /proc/meminfo, GPU count + VRAM from
CUDA). Server parses and caches it, /health exposes as 'hwinfo'. Dashboard
renders it with icons at the top of the Runtime section so the user sees
immediately what the engine is running on.

* web: downgrade to React 18 — React 19 effect cleanup bug crashes the dashboard

React 19 treats the return value of every useEffect callback as a
cleanup function. In practice, any async interaction (streamChat's
rapid onDelta re-renders, the health poller, auto-connect) caused
React 19 to store a Promise as inst.destroy and crash with
'destroy_ is not a function' on the next commit cycle. The bug
reproduced in incognito, without StrictMode, and with every effect
converted to explicit block bodies returning undefined — it is a
React 19 regression in the passive-effect unmount path.

React 18 does not exhibit this behavior. Pin to React 18 until
React 19 is fixed or the codebase migrates to a pattern React 19
handles correctly. All 17 vitest pass, ErrorBoundary retained.

* web: Brain page — the expert cortex, live

A 76x256 canvas grid, one cell per expert (19,456 total): colour =
tier (green VRAM / blue RAM / grey disk), brightness = routing heat
(log-bucketed .coli_usage counts), and a white pulse that flashes on
every expert routed in the current turn and decays — you watch the
model think.

- Engine: EMAP protocol line (1 byte/expert: 2bit tier + 6bit heat,
  hex) at READY and after each turn; HITS bitmap of this turn's routed
  experts (set where eusage increments, cleared on emit).
- Server: parses both, GET /experts serves {rows, cols, map, hits, seq}.
- Web: Brain tab next to Chat; canvas renderer with rAF pulse decay;
  hover tooltip shows layer/expert/tier/heat plus an honest depth-role
  heuristic (early = surface features ... late = output shaping, MTP
  row labelled as the speculative draft head).

* web: Brain page responsive — cells sized from container via ResizeObserver

Cell size derives from the wrapper's actual client box instead of fixed
1400x900, re-rendering on resize; mobile media query tightens padding,
legend and tooltip. The cortex now fills whatever screen it gets.
This commit is contained in:
ZacharyZcR
2026-07-14 22:12:53 +08:00
committed by GitHub
parent d47b095875
commit 2878c60987
11 changed files with 678 additions and 40 deletions
Download Patch File Download Diff File Expand all files Collapse all files
c/coli
+33 -1
View File
@@ -550,6 +550,28 @@ def cmd_serve(a):
a.cap,a.ngen,GLM,env_for(a),a.cors_origin,
a.max_queue,a.queue_timeout,a.kv_slots)
def cmd_web(a):
"""serve + open the dashboard in the browser once the API answers."""
need_model(a.model)
dist = os.path.join(os.path.dirname(os.path.abspath(HERE)), "web", "dist")
if not os.path.exists(os.path.join(dist, "index.html")):
print(f"{C.yel}web UI not built:{C.r} run cd web && npm install && npm run build first;")
print("serving the API anyway (the dashboard will 404 until built).")
url = f"http://{a.host}:{a.port}/"
if not getattr(a, "no_browser", False):
import threading, urllib.request, webbrowser
def opener():
for _ in range(600): # the 744B engine takes minutes to load
time.sleep(2)
try:
urllib.request.urlopen(f"http://{a.host}:{a.port}/health", timeout=2)
webbrowser.open(url); return
except OSError:
continue
threading.Thread(target=opener, daemon=True).start()
print(f"dashboard: {url} (opens automatically when the engine is ready)")
cmd_serve(a)
def cmd_bench(a):
need_model(a.model)
banner("bench")
@@ -620,6 +642,16 @@ def main():
ps.add_argument("--max-queue",type=int,default=int(os.environ.get("COLI_MAX_QUEUE","8")))
ps.add_argument("--queue-timeout",type=float,default=float(os.environ.get("COLI_QUEUE_TIMEOUT","300")))
ps.add_argument("--kv-slots",type=int,default=int(os.environ.get("COLI_KV_SLOTS","1")))
pw=sub.add_parser("web", parents=[common], help="serve + open the dashboard in a browser")
for arg,kw in (("--host",dict(default="127.0.0.1")),("--port",dict(type=int,default=8000)),
("--model-id",dict(default=os.environ.get("COLI_MODEL_ID","glm-5.2-colibri"))),
("--api-key",dict(default=os.environ.get("COLI_API_KEY"))),
("--cors-origin",dict(action="append",default=None)),
("--max-queue",dict(type=int,default=int(os.environ.get("COLI_MAX_QUEUE","8")))),
("--queue-timeout",dict(type=float,default=float(os.environ.get("COLI_QUEUE_TIMEOUT","300")))),
("--kv-slots",dict(type=int,default=int(os.environ.get("COLI_KV_SLOTS","1"))))):
pw.add_argument(arg,**kw)
pw.add_argument("--no-browser",action="store_true",help="don't auto-open the browser")
pb=sub.add_parser("bench", parents=[common]); pb.add_argument("tasks", nargs="*")
pb.add_argument("--limit",type=int,default=40); pb.add_argument("--data",default=os.path.join(HERE,"bench"))
pc=sub.add_parser("convert", parents=[common]); pc.add_argument("--repo",default="zai-org/GLM-5.2-FP8")
@@ -628,7 +660,7 @@ def main():
a=ap.parse_args()
handler={"build":cmd_build,"info":cmd_info,"plan":cmd_plan,"doctor":cmd_doctor,
"run":cmd_run,"chat":cmd_chat,"serve":cmd_serve,"bench":cmd_bench,
"convert":cmd_convert}.get(a.cmd)
"convert":cmd_convert,"web":cmd_web}.get(a.cmd)
if handler: sys.exit(handler(a) or 0)
banner(); print(__doc__)
c/glm.c
+148 -1
View File
@@ -176,7 +176,12 @@ typedef struct {
int64_t resident_bytes;
} Model;
static void usage_save(Model *m); /* cache che impara: definita accanto a stats_dump */
static void usage_save(Model *m);
static void tiers_emit(Model *m);
static void ehit_mark(Model *m, int layer, int eid);
static void emap_emit(Model *m);
static void hits_emit(Model *m);
static void hwinfo_emit(Model *m); /* cache che impara: definita accanto a stats_dump */
#ifdef COLI_CUDA
static int g_cuda_enabled;
static double g_cuda_expert_gb;
@@ -1783,6 +1788,7 @@ static void moe(Model *m, Layer *l, int layer, float *x, int S, float *out){
m->ereq+=keff[s];
for(int kk=0;kk<keff[s];kk++){
m->eusage[layer][idxs[(int64_t)s*K+kk]]++;
ehit_mark(m,layer,idxs[(int64_t)s*K+kk]);
if(m->eheat[layer][idxs[(int64_t)s*K+kk]]<UINT32_MAX) m->eheat[layer][idxs[(int64_t)s*K+kk]]++;
}
for(int d=0;d<D;d++) out[(int64_t)s*D+d]=0;
@@ -1810,6 +1816,7 @@ static void moe(Model *m, Layer *l, int layer, float *x, int S, float *out){
keff[s]=Ke; m->ereq+=Ke;
for(int kk=0;kk<Ke;kk++){
m->eusage[layer][idx[kk]]++;
ehit_mark(m,layer,idx[kk]);
if(m->eheat[layer][idx[kk]]<UINT32_MAX) m->eheat[layer][idx[kk]]++;
m->elast[layer][idx[kk]]=++m->eaccess_clock;
}
@@ -3067,6 +3074,10 @@ static void mux_data(Tok *T, unsigned long long id, int token){
static void mux_done(Model *m, ServeCtx *sc, ServeReq *r){
double dt=now_s()-r->started; if(dt<1e-6) dt=1e-6;
double dh=(double)(m->hits-r->hits0), dm=(double)(m->miss-r->miss0);
hwinfo_emit(m);
tiers_emit(m);
emap_emit(m);
hits_emit(m);
printf("DONE %llu STAT %d %.2f %.1f %.2f %d %d\n",r->id,r->emitted,
r->emitted/dt,(dh+dm)>0?100.0*dh/(dh+dm):0.0,rss_gb(),
r->prompt_tokens,r->length_limited);
@@ -3155,6 +3166,9 @@ static void run_serve_mux(Model *m, const char *snap){
for(int i=0;i<nctx;i++) serve_ctx_init(m,&ctx[i],snap,i,maxctx);
setvbuf(stdin,NULL,_IONBF,0);
printf("\x01\x01READY\x01\x01\nSTAT 0 0.00 0.0 %.2f\n",rss_gb()); fflush(stdout);
hwinfo_emit(m);
tiers_emit(m);
emap_emit(m);
int eof=0;
for(;;){
int active=0; for(int i=0;i<nctx;i++) active+=req[i].active;
@@ -3250,6 +3264,7 @@ static void run_serve(Model *m, const char *snap){
#define first (sc->first)
char *line=NULL; size_t cap=0; ssize_t nr; char *buf=malloc(1<<16);
printf("\x01\x01" "READY" "\x01\x01\n"); printf("STAT 0 0.00 0.0 %.2f\n", rss_gb()); fflush(stdout);
tiers_emit(m);
while((nr=getline(&line,&cap,stdin))>0){
if(nr>0 && line[nr-1]=='\n') line[--nr]=0;
if(!strcmp(line,"\x02RESET")){ len=0; first=1; if(m->has_mtp) m->kv_start[m->c.n_layers]=-1;
@@ -3391,6 +3406,138 @@ static int64_t expert_bytes_probe(Model *m, int ebits){
return eb;
}
/* TIERS: fotografia della piramide expert per la dashboard web —
* "TIERS <vram> <ram> <disk> <vram_gb> <ram_gb>" sul canale di protocollo.
* ram = pinnati non-VRAM + LRU corrente; disk = tutto il resto. */
/* BRAIN MAP (dashboard): per-turn expert hit bitmap + full residency/heat map.
* g_ehit[layer][eid]=1 quando l'expert viene instradato in questo turno;
* hits_emit lo serializza e lo azzera. emap_emit fotografa tier+heat di TUTTI. */
static uint8_t **g_ehit;
static void ehit_mark(Model *m, int layer, int eid){
if(!g_ehit){ Cfg *c=&m->c;
g_ehit=calloc(c->n_layers+1,sizeof(uint8_t*));
for(int i=0;i<=c->n_layers;i++) g_ehit[i]=calloc(c->n_experts,1);
}
g_ehit[layer][eid]=1;
}
/* HWINFO: hardware snapshot for the web dashboard — emitted once at READY. */
static void hwinfo_emit(Model *m){
Cfg *c=&m->c;
/* CPU */
char cpu[256]=""; FILE *ci=fopen("/proc/cpuinfo","r");
if(ci){ char ln[256];
while(fgets(ln,sizeof(ln),ci)) if(!strncmp(ln,"model name",10)){
char *p=strchr(ln,':'); if(p){ p++; while(*p==' ')p++;
int n=(int)strlen(p); if(n>0&&p[n-1]=='\n')p[--n]=0;
snprintf(cpu,sizeof(cpu),"%s",p); } break; }
fclose(ci); }
int cores=0;
#ifdef _SC_NPROCESSORS_ONLN
cores=(int)sysconf(_SC_NPROCESSORS_ONLN);
#endif
/* RAM */
double ram_total=0,ram_avail=0;
FILE *mi=fopen("/proc/meminfo","r");
if(mi){ char ln[256]; double mt=0,ma=0;
while(fgets(ln,sizeof(ln),mi)){
if(sscanf(ln,"MemTotal: %lf",&mt)==1) ram_total=mt/1e6;
if(sscanf(ln,"MemAvailable: %lf",&ma)==1) ram_avail=ma/1e6;
} fclose(mi); }
/* GPU */
int ngpu=0; double vram_total=0;
char gpu_name[128]="";
#ifdef COLI_CUDA
ngpu=g_cuda_ndev; vram_total=m->gpu_expert_bytes/1e9;
for(int i=0;i<g_cuda_ndev;i++){
size_t fr=0,to=0; coli_cuda_mem_info(g_cuda_devices[i],&fr,&to);
if(!i) vram_total=(double)to*g_cuda_ndev/1e9;
}
/* GPU name from the first device — already printed at init */
if(g_cuda_ndev>0){
/* We don't have a device-name API; parse from the init log line stored in stderr.
* Simpler: just read the nvidia driver sysfs or use a fixed label. */
snprintf(gpu_name,sizeof(gpu_name),"CUDA device x%d",g_cuda_ndev);
}
#endif
printf("HWINFO %d %.1f %.1f %d %.1f %s|%s\n",
cores,ram_total,ram_avail,ngpu,vram_total,cpu,gpu_name);
fflush(stdout);
}
static void tiers_emit(Model *m){
Cfg *c=&m->c; int nsp=0;
for(int i=0;i<c->n_layers;i++) if(m->L[i].sparse) nsp++;
int total=(nsp+(m->has_mtp?1:0))*c->n_experts;
int pinned=0,lru=0;
for(int i=0;i<=c->n_layers;i++){ pinned+=m->npin?m->npin[i]:0; lru+=m->ecn?m->ecn[i]:0; }
int vram=0; double vram_gb=0;
#ifdef COLI_CUDA
vram=m->gpu_expert_count; vram_gb=m->gpu_expert_bytes/1e9;
#endif
int ram=pinned-vram+lru; if(ram<0) ram=0;
int disk=total-vram-ram; if(disk<0) disk=0;
double eb=(double)expert_bytes_probe(m,m->ebits);
printf("TIERS %d %d %d %.2f %.2f\n",vram,ram,disk,vram_gb,ram*eb/1e9);
fflush(stdout);
}
/* EMAP: 1 byte/expert (2bit tier: 0=disk 1=RAM 2=VRAM | 6bit heat log2-bucket),
* righe = layer sparsi in ordine (+MTP se presente), colonne = n_experts. Hex. */
static void emap_emit(Model *m){
Cfg *c=&m->c;
int rows=0;
for(int i=0;i<c->n_layers;i++) if(m->L[i].sparse) rows++;
int has_mtp = m->has_mtp && m->eusage[c->n_layers];
if(has_mtp) rows++;
int cols=c->n_experts;
char *hex=malloc((size_t)rows*cols*2+1); int w=0;
for(int i=0;i<=c->n_layers;i++){
int is_row = (i<c->n_layers && m->L[i].sparse) || (i==c->n_layers && has_mtp);
if(!is_row) continue;
for(int e=0;e<cols;e++){
int tier=0;
ESlot *P=m->pin[i];
for(int z=0;z<m->npin[i];z++) if(P[z].eid==e){
#ifdef COLI_CUDA
tier = P[z].g.cuda?2:1;
#else
tier = 1;
#endif
break; }
if(!tier && m->ecache && m->ecache[i])
for(int z=0;z<m->ecn[i];z++) if(m->ecache[i][z].eid==e){ tier=1; break; }
uint32_t u = m->eusage[i]?m->eusage[i][e]:0;
int heat=0; while(u){ heat++; u>>=1; } if(heat>63) heat=63;
int b=(tier<<6)|heat;
hex[w++]="0123456789abcdef"[b>>4]; hex[w++]="0123456789abcdef"[b&15];
}
}
hex[w]=0;
printf("EMAP %d %d %s\n",rows,cols,hex); fflush(stdout); free(hex);
}
/* HITS: bitmap 1bit/expert (stesso ordine di EMAP), poi azzera per il turno dopo. */
static void hits_emit(Model *m){
Cfg *c=&m->c; if(!g_ehit) return;
int rows=0;
for(int i=0;i<c->n_layers;i++) if(m->L[i].sparse) rows++;
int has_mtp = m->has_mtp && m->eusage[c->n_layers];
if(has_mtp) rows++;
int cols=c->n_experts, nb=(rows*cols+7)/8;
uint8_t *bm=calloc(nb,1); int bit=0;
for(int i=0;i<=c->n_layers;i++){
int is_row = (i<c->n_layers && m->L[i].sparse) || (i==c->n_layers && has_mtp);
if(!is_row) continue;
for(int e=0;e<cols;e++,bit++)
if(g_ehit[i][e]){ bm[bit>>3]|=1<<(bit&7); g_ehit[i][e]=0; }
}
char *hex=malloc((size_t)nb*2+1); int w=0;
for(int b=0;b<nb;b++){ hex[w++]="0123456789abcdef"[bm[b]>>4]; hex[w++]="0123456789abcdef"[bm[b]&15]; }
hex[w]=0;
printf("HITS %d %d %s\n",rows,cols,hex); fflush(stdout); free(hex); free(bm);
}
/* scarica su file l'istogramma d'uso degli expert: righe "layer eid count" (per PIN).
* Include la riga MTP (layer n_layers). Scrittura atomica (tmp+rename): viene chiamata
* anche a ogni turno di serve e il processo puo' morire in qualsiasi momento. */
c/openai_server.py
+71 -2
View File
@@ -7,6 +7,7 @@ import collections
import contextlib
import json
import math
import mimetypes
import os
import select
import queue
@@ -27,6 +28,8 @@ END = b"\x01\x01END\x01\x01\n"
READY = b"\x01\x01READY\x01\x01\n"
MAX_BODY = 4 << 20
DEFAULT_CORS_ORIGINS = (
"http://127.0.0.1:8000",
"http://localhost:8000",
"http://127.0.0.1:5173",
"http://localhost:5173",
"http://tauri.localhost",
@@ -462,6 +465,11 @@ class Engine:
self.closed = False
self.dispatcher_error = None
self.kv_slots = kv_slots
self.tiers = None
self.hwinfo = None
self.emap = None
self.hits = None
self.hits_seq = 0 # latest "TIERS" snapshot from the engine
read_engine_turn(self.process.stdout, READY, lambda _: None)
self.dispatcher = threading.Thread(target=self._dispatch_stdout,
name="colibri-stdout", daemon=True)
@@ -527,6 +535,22 @@ class Engine:
events = self.pending.pop(request_id, None)
if events is not None:
events.put(("done", stats))
elif kind == "HWINFO" and len(fields) >= 7:
parts = " ".join(fields[6:]).split("|")
self.hwinfo = {"cores": int(fields[1]), "ram_total_gb": float(fields[2]),
"ram_avail_gb": float(fields[3]), "gpus": int(fields[4]),
"vram_total_gb": float(fields[5]),
"cpu": parts[0].strip() if len(parts)>0 else "",
"gpu": parts[1].strip() if len(parts)>1 else ""}
elif kind == "EMAP" and len(fields) == 4:
self.emap = {"rows": int(fields[1]), "cols": int(fields[2]), "map": fields[3]}
elif kind == "HITS" and len(fields) == 4:
self.hits = fields[3]
self.hits_seq += 1
elif kind == "TIERS" and len(fields) >= 6:
self.tiers = {"vram": int(fields[1]), "ram": int(fields[2]),
"disk": int(fields[3]), "vram_gb": float(fields[4]),
"ram_gb": float(fields[5])}
elif kind == "ERROR" and len(fields) >= 2:
request_id = fields[1]
message = " ".join(fields[2:]) or "engine request failed"
@@ -700,13 +724,58 @@ class APIHandler(BaseHTTPRequestHandler):
if model != self.server.model_id:
raise APIError(404, f"The model `{model}` does not exist.", "model", "model_not_found")
WEB_DIST = Path(__file__).resolve().parent.parent / "web" / "dist"
def serve_static(self, path):
"""Serve the built web UI (web/dist) so `coli web` is one process.
Read-only, no auth (same trust level as /health), traversal-safe."""
if path.startswith("/v1/") or path == "/health":
return False
base = self.WEB_DIST
if not base.is_dir():
return False
rel = unquote(path).lstrip("/") or "index.html"
target = (base / rel).resolve()
if not str(target).startswith(str(base)) or not target.is_file():
if path == "/" or "." not in rel: # SPA fallback
target = base / "index.html"
if not target.is_file():
return False
else:
return False
ctype = mimetypes.guess_type(str(target))[0] or "application/octet-stream"
data = target.read_bytes()
self.send_response(200)
self.send_header("Content-Type", ctype)
self.send_header("Content-Length", str(len(data)))
self.send_cors_headers()
self.end_headers()
self.wfile.write(data)
return True
def do_GET(self):
request_id = "req_" + uuid.uuid4().hex
try:
path = urlsplit(self.path).path
if path == "/health":
self.send_json(200, {"status": "ok", "scheduler": self.server.scheduler.snapshot(),
"kv_slots": self.server.kv_slots}, request_id)
payload = {"status": "ok", "scheduler": self.server.scheduler.snapshot(),
"kv_slots": self.server.kv_slots}
tiers = getattr(self.server.engine, "tiers", None) if self.server.engine else None
if tiers: payload["tiers"] = tiers
hwinfo = getattr(self.server.engine, "hwinfo", None) if self.server.engine else None
if hwinfo: payload["hwinfo"] = hwinfo
self.send_json(200, payload, request_id)
return
if path == "/experts":
eng = self.server.engine
payload = {"rows": 0, "cols": 0, "map": "", "hits": "", "seq": 0}
if eng and getattr(eng, "emap", None):
payload.update(eng.emap)
payload["hits"] = eng.hits or ""
payload["seq"] = eng.hits_seq
self.send_json(200, payload, request_id)
return
if self.serve_static(path):
return
self.require_auth()
if path == "/v1/models":