Merge pull request #391 from ZacharyZcR/refactor/split-colibri
refactor: split glm.c → colibri.c + 4 header modules (−18%)
This commit is contained in:
Vincenzo Fornaro
@@ -12,8 +12,8 @@ jobs:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Build glm
|
||||
run: cd c && make glm
|
||||
- name: Build colibri
|
||||
run: cd c && make colibri
|
||||
- name: C test suite
|
||||
run: cd c && make test-c
|
||||
|
||||
@@ -105,13 +105,13 @@ jobs:
|
||||
make cuda-dll CUDA_ARCH=sm_80
|
||||
test -f coli_cuda.dll || { echo "cuda-dll reported success but produced no DLL" >&2; exit 1; }
|
||||
echo "coli_cuda.dll built (MSVC host)"
|
||||
- name: make glm CUDA_DLL=1 (host links backend_loader, not cudart)
|
||||
- name: make colibri CUDA_DLL=1 (host links backend_loader, not cudart)
|
||||
shell: msys2 {0}
|
||||
run: |
|
||||
cd c
|
||||
make glm CUDA_DLL=1
|
||||
test -f glm.exe || { echo "glm CUDA_DLL=1 reported success but produced no exe" >&2; exit 1; }
|
||||
echo "glm.exe built against the DLL loader"
|
||||
make colibri CUDA_DLL=1
|
||||
test -f colibri.exe || { echo "colibri CUDA_DLL=1 reported success but produced no exe" >&2; exit 1; }
|
||||
echo "colibri.exe built against the DLL loader"
|
||||
|
||||
web:
|
||||
name: Web UI
|
||||
|
||||
@@ -10,6 +10,8 @@ desktop/src-tauri/target/
|
||||
desktop/src-tauri/gen/
|
||||
|
||||
# binari compilati (si rigenerano con make / coli build)
|
||||
c/colibri
|
||||
c/colibri.exe
|
||||
c/glm
|
||||
c/glm.exe
|
||||
c/olmoe
|
||||
|
||||
+260
@@ -0,0 +1,260 @@
|
||||
<p align="center">
|
||||
<img src="assets/colibri.svg" width="500" alt="colibrì — motore piccolo, modello immenso">
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="README.md">English</a> · <a href="README.zh-CN.md">简体中文</a> · <a href="README.zh-TW.md">繁體中文</a> · Italiano
|
||||
</p>
|
||||
|
||||
**Motore piccolo, modello immenso.** Esegui **GLM-5.2 (744 miliardi di parametri, MoE)** su un computer consumer con ~25 GB di RAM — in C puro, zero dipendenze, caricando gli expert dal disco in streaming.
|
||||
|
||||
Colibrì è un runtime MoE leggero e che preserva la qualità: tratta VRAM, RAM e
|
||||
disco come un'unica gerarchia di memoria gestita. Se la memoria veloce non basta
|
||||
il modello rallenta, ma la policy predefinita **non cambia mai silenziosamente la
|
||||
precisione del modello né la semantica del router**.
|
||||
|
||||
```
|
||||
$ ./coli chat
|
||||
🐦 colibrì v1.0 — GLM-5.2 · 744B MoE · int4 · streaming CPU
|
||||
✓ ready in 32s · resident 9.9 GB
|
||||
› ciao!
|
||||
◆ Ciao! 😊 Come posso aiutarti oggi?
|
||||
```
|
||||
|
||||
## Guardalo in azione
|
||||
|
||||
<p align="center">
|
||||
<img src="docs/media/colibri-dashboard.png" width="900" alt="dashboard web di colibrì — metriche live, pannello hardware, livelli degli expert">
|
||||
</p>
|
||||
<p align="center"><em>La dashboard web (<code>./coli web</code>): un modello da 744B a <strong>4 tok/s, TTFT 1.6 s, disco 0</strong> —
|
||||
residenza completa degli expert su 6× RTX 5090, con metriche token in tempo reale, breakdown dei tempi per turno,
|
||||
la barra dei livelli VRAM/RAM/disco e il mini-cervello live nell'angolo.</em></p>
|
||||
|
||||
<p align="center">
|
||||
<img src="docs/media/colibri-brain.png" width="900" alt="la pagina Brain — 19.456 expert come una corteccia vivente">
|
||||
</p>
|
||||
<p align="center"><em>La pagina <strong>Brain</strong>: tutti i 19.456 expert come una corteccia vivente — il colore indica
|
||||
il livello di archiviazione, la luminosità il calore di routing, e ogni expert instradato in un turno
|
||||
lampeggia bianco. Passando il cursore si vede l'<a href="https://github.com/JustVugg/colibri/issues/175">affinità
|
||||
tematica misurata</a> dell'expert.</em></p>
|
||||
|
||||
<p align="center">
|
||||
<img src="docs/media/colibri-atlas.png" width="900" alt="la pagina Atlas — l'atlante misurato degli expert come una galassia 3D">
|
||||
</p>
|
||||
<p align="center"><em>La pagina <strong>Atlas</strong>: l'<a href="https://github.com/JustVugg/colibri/issues/175">atlante
|
||||
misurato degli expert</a> come una galassia 3D — 13.260 expert caratterizzati, 1.041 specialisti
|
||||
replicabili che si raggruppano per argomento (poesia, legge, cinese, SQL…). La posizione deriva
|
||||
dall'affinità di routing misurata, non da un embedding appreso. Trascinare per ruotare.</em></p>
|
||||
|
||||
## L'idea
|
||||
|
||||
Un modello Mixture-of-Experts da 744B attiva solo ~40B parametri per token — e
|
||||
solo ~11 GB di quelli cambiano da un token all'altro (gli expert instradati):
|
||||
|
||||
<p align="center">
|
||||
<img src="docs/media/sparse.png" width="880" alt="solo ~5.4% dei parametri è attivo per token">
|
||||
</p>
|
||||
|
||||
Il modello non ha bisogno di *stare* in memoria veloce — ha bisogno di essere
|
||||
**piazzato**:
|
||||
|
||||
- la **parte densa** (attenzione, expert condivisi, embedding — ~17B parametri)
|
||||
resta **residente in RAM a int4** (~9.9 GB);
|
||||
- i **19.456 expert instradati** (75 layer MoE × 256 + la testa MTP, ~19 MB
|
||||
ciascuno a int4) stanno **su disco** (~370 GB) e vengono **caricati on demand
|
||||
in streaming**, con una cache LRU per layer, un hot-store pinnato che impara,
|
||||
e un livello VRAM opzionale.
|
||||
|
||||
Il motore è un singolo file C (`c/colibri.c`) più header piccoli. Niente BLAS,
|
||||
niente Python a runtime, niente GPU obbligatoria.
|
||||
|
||||
## Come funziona
|
||||
|
||||
### Il percorso di ogni token
|
||||
|
||||
<p align="center">
|
||||
<img src="docs/media/token-path.png" width="880" alt="instrada → unione → piazza → sovrapponi → impara">
|
||||
</p>
|
||||
|
||||
Ogni layer di ogni token percorre gli stessi cinque passi. L'obiettivo
|
||||
progettuale è che **il piazzamento decide solo la velocità** — le decisioni
|
||||
del router e la precisione dei pesi sono identiche sia che un expert risponda
|
||||
dalla VRAM sia dal disco.
|
||||
|
||||
### Una gerarchia di memoria, non un requisito di memoria
|
||||
|
||||
<p align="center">
|
||||
<img src="docs/media/tiers.png" width="880" alt="residenza expert a tre livelli: VRAM / RAM / NVMe">
|
||||
</p>
|
||||
|
||||
Lo stesso motore copre l'intero spettro: su un portatile da 25 GB tutto viene
|
||||
caricato dal disco in streaming (lento, ma corretto); su un host grande l'intero
|
||||
set di expert diventa residente (`CUDA_EXPERT_GB=auto PIN_GB=all`) e il disco
|
||||
esce completamente dal percorso di decode. Tra i livelli c'è una **cache che
|
||||
impara**: il motore registra quali expert il *tuo* carico di lavoro instrada
|
||||
(`.coli_usage`, aggiornato a ogni turno) e fissa automaticamente i più caldi —
|
||||
colibrì diventa letteralmente più veloce man mano che lo usi. Sugli host
|
||||
multi-socket, `COLI_NUMA=1` interlaccia i pesi residenti tra i controller di
|
||||
memoria ([#82](https://github.com/JustVugg/colibri/issues/82)).
|
||||
|
||||
### Mai aspettare il disco due volte
|
||||
|
||||
I miss nella cache costano caro, quindi il motore investe la maggior parte
|
||||
della sua astuzia per evitarli e sovrapporli: le tre matrici di ogni expert sono
|
||||
memorizzate contigue e lette con un unico `pread`; un pool I/O asincrono
|
||||
limitato (`PIPE=1`, attivo per default) carica gli expert mancanti mentre quelli
|
||||
residenti calcolano; le posizioni in batch leggono ogni expert unico una sola
|
||||
volta (**batch-union**); un thread di lookahead del router (`PILOT=1`) fa il
|
||||
prefetch degli expert del layer successivo — il routing è misurabilmente
|
||||
**prevedibile al 71.6% un layer in anticipo**. Sulle GPU, la pipeline residente
|
||||
(`COLI_CUDA_PIPE=2`) mantiene il flusso residuo on-device tra i layer, così il
|
||||
loop CPU degli expert procede senza interruzioni; su Apple Silicon un backend
|
||||
[Metal](docs/metal.md) sperimentale esegue la matmul batch degli expert sulla
|
||||
GPU a memoria unificata.
|
||||
|
||||
### Modello fedele, stato compresso
|
||||
|
||||
Il forward pass è validato **token-esatto contro un oracle `transformers`**
|
||||
(teacher-forcing 32/32). L'attenzione MLA memorizza uno stato KV compresso — 576
|
||||
float/token invece di 32.768 (**57× più piccolo**) — e lo persiste tra i
|
||||
riavvii (`.coli_kv`): le conversazioni riaprono "calde", senza alcun re-prefill,
|
||||
byte-identiche a una sessione ininterrotta. L'attenzione sparsa DSA (il
|
||||
lightning indexer di GLM-5.2) è implementata fedelmente e validata forzando la
|
||||
selezione di tutte le chiavi per riprodurre esattamente l'attenzione densa.
|
||||
|
||||
### Decodifica speculativa, onestamente
|
||||
|
||||
La testa MTP nativa di GLM-5.2 propone token che il modello principale verifica
|
||||
in un unico forward batch — 2.2–2.8 token/forward quando conviene. Due regole
|
||||
conquistate a caro prezzo sono i default: la testa MTP deve essere **int8** (le
|
||||
teste int4 crollano al 0–4% di accettazione,
|
||||
[#8](https://github.com/JustVugg/colibri/issues/8)), e draft e verifica devono
|
||||
calcolare **la stessa funzione** — `SPEC_PIN=1` fissa entrambi sulla stessa
|
||||
famiglia di kernel ([#163](https://github.com/JustVugg/colibri/issues/163)
|
||||
contiene l'intera indagine forense). I draft forzati da grammatica
|
||||
([`GRAMMAR=file.gbnf`](docs/grammar-draft.md)) aggiungono accettazione quasi
|
||||
gratuita sull'output JSON vincolato. Se la speculazione conviene dipende dalla
|
||||
temperatura della cache — misura, e usa `DRAFT=0` quando non paga.
|
||||
|
||||
## Cosa ottiene
|
||||
|
||||
<p align="center">
|
||||
<img src="docs/media/ladder.png" width="880" alt="velocità di decode misurata per classe hardware">
|
||||
</p>
|
||||
|
||||
Stesso motore, stesso container int4 — cambia solo dove risiedono gli expert.
|
||||
Punti salienti dalle [tabelle benchmark complete](docs/benchmarks.md):
|
||||
|
||||
- **6× RTX 5090, residenza completa:** 5.8–6.8 tok/s in decode, TTFT ~13 s
|
||||
([log dell'esperimento](docs/experiments/glm52-6x5090-2026-07-12.md));
|
||||
- **desktop solo-CPU da 128 GB:** ~1.8 tok/s a cache calda
|
||||
([#200](https://github.com/JustVugg/colibri/issues/200));
|
||||
- **singola RTX 5070 Ti, classe laptop:** 1.07 tok/s tramite la pipeline
|
||||
GPU-residente ([#273](https://github.com/JustVugg/colibri/issues/273));
|
||||
- **macchina di sviluppo da 25 GB:** 0.05–0.1 tok/s a freddo — il punto di
|
||||
partenza dimostrato da cui è nato il progetto, e ancora oggi la baseline onesta.
|
||||
|
||||
La qualità è misurata, non presunta: il costo di quantizzazione del container
|
||||
int4 e le ablazioni su granularità delle scale e rotazione sono in
|
||||
[docs/benchmarks.md](docs/benchmarks.md#quality-benchmark) e
|
||||
[#108](https://github.com/JustVugg/colibri/issues/108)/[#81](https://github.com/JustVugg/colibri/issues/81).
|
||||
|
||||
## Per iniziare
|
||||
|
||||
### 1. Scarica il modello
|
||||
|
||||
Un container **GLM-5.2 int4** pre-convertito è su Hugging Face — **usa la
|
||||
versione con le teste MTP int8**:
|
||||
|
||||
**https://huggingface.co/mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp**
|
||||
|
||||
> ⚠️ Il mirror originale contiene teste MTP int4 → accettazione dei draft allo 0%
|
||||
> ([#8](https://github.com/JustVugg/colibri/issues/8)). Verifica la tua versione:
|
||||
> `ls -l <modello>/out-mtp-*` — int8 (corretto) è `3527131672 / 5366238584 / 1065950496`.
|
||||
|
||||
Oppure converti tu stesso dalla sorgente FP8 — un unico comando riprendibile che
|
||||
non richiede mai i 756 GB completi su disco contemporaneamente:
|
||||
|
||||
```bash
|
||||
cd c && ./setup.sh # verifica gcc/OpenMP, compila, autotest
|
||||
./coli convert --model /nvme/glm52_i4 # scarica e converti shard per shard (python, una tantum)
|
||||
```
|
||||
|
||||
### 2. Esegui
|
||||
|
||||
```bash
|
||||
COLI_MODEL=/nvme/glm52_i4 ./coli chat # budget RAM, cache e MTP rilevati automaticamente
|
||||
COLI_MODEL=/nvme/glm52_i4 ./coli plan # mostra il piazzamento pianificato VRAM/RAM/disco
|
||||
COLI_MODEL=/nvme/glm52_i4 ./coli doctor # controllo di idoneità (sola lettura)
|
||||
./coli web --model /nvme/glm52_i4 # API + dashboard web sulla stessa porta
|
||||
./coli serve --model /nvme/glm52_i4 # solo API compatibile OpenAI
|
||||
```
|
||||
|
||||
Il motore a runtime è puro C — python si usa solo per il convertitore (una tantum)
|
||||
e per il gateway API opzionale.
|
||||
|
||||
### 3. Approfondisci
|
||||
|
||||
| argomento | documento |
|
||||
|---|---|
|
||||
| Benchmark, dati dalla comunità, misurazioni di qualità | [docs/benchmarks.md](docs/benchmarks.md) |
|
||||
| Parametri di tuning, policy, cache che impara, prefetch | [docs/tuning.md](docs/tuning.md) |
|
||||
| Build nativa su Windows 11 (con CUDA DLL) | [docs/windows.md](docs/windows.md) |
|
||||
| Backend CUDA, livello expert in VRAM, residenza completa | [docs/cuda.md](docs/cuda.md) |
|
||||
| Backend Metal per Apple Silicon | [docs/metal.md](docs/metal.md) |
|
||||
| API compatibile OpenAI, KV slot, dashboard web | [docs/api.md](docs/api.md) |
|
||||
| Draft forzati da grammatica (output strutturato) | [docs/grammar-draft.md](docs/grammar-draft.md) |
|
||||
| Inventario delle variabili d'ambiente | [docs/ENVIRONMENT.md](docs/ENVIRONMENT.md) |
|
||||
|
||||
## Sostenere il progetto
|
||||
|
||||
colibrì è nato come progetto di una sola persona su un portatile con 12 core
|
||||
e 25 GB di RAM; oggi i suoi numeri arrivano da una comunità di macchine reali.
|
||||
Se ti è utile:
|
||||
|
||||
- ⭐ metti una stella al repository e condividilo;
|
||||
- 🐛 apri issue con i numeri di benchmark del tuo hardware — i datapoint
|
||||
fanno avanzare questo progetto più di qualsiasi altra cosa;
|
||||
- 💬 contattaci via GitHub issues per sponsorizzare lo sviluppo o donare hardware.
|
||||
|
||||
## Struttura del repository
|
||||
|
||||
```
|
||||
Makefile punto d'ingresso root per build/check
|
||||
c/
|
||||
├── colibri.c motore principale
|
||||
├── quant.h kernel matmul quantizzati (SIMD multi-architettura)
|
||||
├── sample.h campionamento, RNG, set di stop
|
||||
├── kv_persist.h persistenza KV su disco (.coli_kv)
|
||||
├── telemetry.h protocollo dashboard, statistiche, usage
|
||||
├── st.h, tok.h, json.h header di runtime
|
||||
├── backend_cuda.* livello CUDA opzionale
|
||||
├── Makefile build e check locali
|
||||
├── coli CLI utente
|
||||
├── openai_server.py gateway HTTP compatibile OpenAI
|
||||
├── setup.sh setup locale in un solo comando
|
||||
├── tools/ conversione offline, fixture e benchmark
|
||||
├── scripts/ helper per conversioni lunghe
|
||||
└── tests/ test C e Python senza dipendenze
|
||||
web/ UI browser (puro client API OpenAI)
|
||||
desktop/ shell desktop Tauri v2 che racchiude la web UI
|
||||
docs/ documentazione di riferimento, esperimenti, media
|
||||
```
|
||||
|
||||
Il percorso a runtime resta intenzionalmente piatto e leggibile: `colibri.c`
|
||||
più i suoi header. Dalla radice del repository, `make`, `make check` e
|
||||
`make clean` delegano al Makefile del motore.
|
||||
|
||||
## Perché "colibrì"
|
||||
|
||||
Il colibrì pesa pochi grammi, sta sospeso nel vuoto e visita un migliaio di
|
||||
fiori al giorno. Questo motore tiene in vita un gigante da 744 miliardi di
|
||||
parametri con le razioni di un colibrì: 25 GB di RAM, dodici core CPU e
|
||||
tanta pazienza col disco.
|
||||
|
||||
Il nome è rimasto in italiano perché questa è la lingua in cui è stato scritto
|
||||
il primo prototipo — i commenti nel codice lo testimoniano ancora.
|
||||
|
||||
## Licenza
|
||||
|
||||
Apache 2.0. I pesi di GLM-5.2 sono rilasciati da Z.ai sotto licenza MIT.
|
||||
@@ -3,7 +3,7 @@
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
English · <a href="README.zh-TW.md">繁體中文</a>
|
||||
English · <a href="README.zh-CN.md">简体中文</a> · <a href="README.zh-TW.md">繁體中文</a> · <a href="README.it.md">Italiano</a>
|
||||
</p>
|
||||
|
||||
**Tiny engine, immense model.** Run **GLM-5.2 (744B-parameter MoE)** on a consumer machine with ~25 GB of RAM — in pure C, with zero dependencies, by streaming experts from disk.
|
||||
|
||||
+237
@@ -0,0 +1,237 @@
|
||||
<p align="center">
|
||||
<img src="assets/colibri.svg" width="500" alt="colibrì——小巧引擎，庞大模型">
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="README.md">English</a> · 简体中文 · <a href="README.zh-TW.md">繁體中文</a> · <a href="README.it.md">Italiano</a>
|
||||
</p>
|
||||
|
||||
**小巧引擎，庞大模型。**只需约 25 GB 内存，就能在消费级电脑上运行 **GLM-5.2（744B 参数的 MoE）**——以零依赖的纯 C 实现，从磁盘流式加载专家。
|
||||
|
||||
Colibrì 是一套轻量、保持模型质量的 MoE 运行时，将 VRAM、RAM
|
||||
与存储设备视为统一管理的内存层级。高速内存不足可能降低速度，
|
||||
但默认策略**绝不会在未告知的情况下改变模型精度或路由语义**。
|
||||
|
||||
```
|
||||
$ ./coli chat
|
||||
🐦 colibrì v1.0 — GLM-5.2 · 744B MoE · int4 · streaming CPU
|
||||
✓ ready in 32s · resident 9.9 GB
|
||||
› ciao!
|
||||
◆ Ciao! 😊 Come posso aiutarti oggi?
|
||||
```
|
||||
|
||||
## 实际运行效果
|
||||
|
||||
<p align="center">
|
||||
<img src="docs/media/colibri-dashboard.png" width="900" alt="colibrì 网页仪表盘——实时指标、硬件面板与专家存储层级">
|
||||
</p>
|
||||
<p align="center"><em>网页仪表盘（<code>./coli web</code>）：744B 模型达到 <strong>4 tok/s、TTFT 1.6 秒、磁盘读取 0</strong>——
|
||||
在 6× RTX 5090 上让所有专家常驻，并实时显示 token 指标、每轮耗时明细、
|
||||
VRAM／RAM／磁盘层级条，以及角落的实时迷你大脑。</em></p>
|
||||
|
||||
<p align="center">
|
||||
<img src="docs/media/colibri-brain.png" width="900" alt="大脑页面——以实时皮层呈现 19,456 个专家">
|
||||
</p>
|
||||
<p align="center"><em><strong>大脑（Brain）</strong>页面：将全部 19,456 个专家呈现为活的皮层——颜色代表存储层级，
|
||||
亮度代表路由热度，每轮被路由到的专家都会闪白。将光标停在专家上，即可查看其
|
||||
<a href="https://github.com/JustVugg/colibri/issues/175">实测主题亲和度</a>。</em></p>
|
||||
|
||||
<p align="center">
|
||||
<img src="docs/media/colibri-atlas.png" width="900" alt="图谱页面——以 3D 星系呈现实测专家图谱">
|
||||
</p>
|
||||
<p align="center"><em><strong>图谱（Atlas）</strong>页面：将<a href="https://github.com/JustVugg/colibri/issues/175">实测专家图谱</a>
|
||||
呈现为 3D 星系——共 13,260 个已分析专家，其中 1,041 个可复现的专门专家会按主题聚集
|
||||
（诗歌、法律、中文、SQL……）。位置取自实测路由亲和度，而非学习出的嵌入向量。拖拽即可旋转。</em></p>
|
||||
|
||||
## 核心概念
|
||||
|
||||
744B 的专家混合（Mixture-of-Experts）模型，每个 token 只会激活约 40B 参数——
|
||||
其中每个 token 之间会变动的只有约 11 GB（被路由到的专家）：
|
||||
|
||||
<p align="center">
|
||||
<img src="docs/media/sparse.png" width="880" alt="每个 token 只会激活约 5.4% 的参数">
|
||||
</p>
|
||||
|
||||
所以模型不必完整**装进**高速内存，而是需要正确**放置**：
|
||||
|
||||
- **稠密部分**（注意力、共享专家、嵌入——约 17B 参数）以 int4
|
||||
**常驻 RAM**（约 9.9 GB）；
|
||||
- **19,456 个路由专家**（75 个 MoE 层 × 256，加上 MTP head；每个在 int4 下约 19 MB）
|
||||
**存放在磁盘**（约 370 GB），并**按需流式加载**，配合逐层 LRU 缓存、
|
||||
会学习的热门专家固定存储区，以及可选的 VRAM 层级。
|
||||
|
||||
引擎是一个 C 主文件（`c/colibri.c`）加上若干头文件。不需要 BLAS，
|
||||
运行时不需要 Python，也不需要 GPU。
|
||||
|
||||
## 工作原理
|
||||
|
||||
### 每个 token 的处理路径
|
||||
|
||||
<p align="center">
|
||||
<img src="docs/media/token-path.png" width="880" alt="路由 → 并集 → 放置 → 重叠执行 → 学习">
|
||||
</p>
|
||||
|
||||
每个 token 的每一层都会经过相同的五个步骤。设计目标是让
|
||||
**放置只决定速度**——无论专家是从 VRAM 还是磁盘响应，路由器的决策与权重精度都完全相同。
|
||||
|
||||
### 统一内存层级，取代单一内存门槛
|
||||
|
||||
<p align="center">
|
||||
<img src="docs/media/tiers.png" width="880" alt="VRAM／RAM／NVMe 三层专家常驻架构">
|
||||
</p>
|
||||
|
||||
同一套引擎覆盖完整硬件范围：在 25 GB 笔记本上，一切都从磁盘流式加载
|
||||
（慢，但结果正确）；在大内存主机上，则可让整组专家常驻
|
||||
（`CUDA_EXPERT_GB=auto PIN_GB=all`），让磁盘完全退出解码路径。
|
||||
两端之间有一层**学习型缓存**：引擎会记录*你的*工作负载路由到哪些专家
|
||||
（`.coli_usage`，每轮更新），并自动固定最热门的专家——colibrì 确实会越用越快。
|
||||
在多路主机上，`COLI_NUMA=1` 会将常驻权重交错分配到各内存控制器
|
||||
（[#82](https://github.com/JustVugg/colibri/issues/82)）。
|
||||
|
||||
### 绝不为同一次磁盘读取等待两遍
|
||||
|
||||
缓存未命中的代价很高，因此引擎大部分的巧思都用来避免或重叠这些读取：
|
||||
每个专家的三个矩阵相邻存储，并以一次 `pread` 读取；有界异步 I/O 池
|
||||
（`PIPE=1`，默认启用）会在常驻专家计算时加载缺失的专家；批量位置只读取每个
|
||||
不重复专家一次（**批量并集**）；路由前瞻线程（`PILOT=1`）则预取下一层专家——
|
||||
实测显示，路由结果提前一层时有 **71.6% 的可预测性**。
|
||||
在 GPU 上，常驻管线（`COLI_CUDA_PIPE=2`）让残差流跨层保留在设备端，
|
||||
使 CPU 专家循环不中断；在 Apple Silicon 上，实验性的
|
||||
[Metal 后端](docs/metal.md)会用统一内存 GPU 执行批量专家运算。
|
||||
|
||||
### 忠实模型，压缩状态
|
||||
|
||||
前向传播已通过 `transformers` oracle 验证为**逐 token 完全一致**
|
||||
（teacher-forcing 32/32）。MLA 注意力存储压缩后的 KV 状态——每个 token 为 576 个
|
||||
浮点数，而非 32,768 个（**缩小 57×**）——并跨重启持久保存
|
||||
（`.coli_kv`）：对话可暖启恢复，不需重新 prefill，结果与不中断的会话
|
||||
逐字节相同。DSA 稀疏注意力（GLM-5.2 的 lightning indexer）已忠实实现，
|
||||
并通过强制选取所有 key，验证可精确复现稠密注意力。
|
||||
|
||||
### 诚实的推测解码
|
||||
|
||||
GLM-5.2 原生 MTP head 会起草 token，再由主模型以一次批量前向传播验证——
|
||||
条件合适时每次 forward 可产生 2.2–2.8 个 token。两条来之不易的规则已成为默认值：
|
||||
MTP head 必须是 **int8**（int4 head 的接受率会崩塌到 0–4%，见
|
||||
[#8](https://github.com/JustVugg/colibri/issues/8)），且草稿与验证必须计算
|
||||
**相同函数**——`SPEC_PIN=1` 会把两者固定在同一 kernel family
|
||||
（完整取证过程见 [#163](https://github.com/JustVugg/colibri/issues/163)）。
|
||||
语法强制草稿（[`GRAMMAR=file.gbnf`](docs/grammar-draft.md)）可在受限 JSON 输出中，
|
||||
以近乎免费的代价提高接受率。推测解码是否带来净收益取决于缓存热度——请实测，
|
||||
若不划算就使用 `DRAFT=0`。
|
||||
|
||||
## 实际成果
|
||||
|
||||
<p align="center">
|
||||
<img src="docs/media/ladder.png" width="880" alt="各硬件级别的实测解码速度">
|
||||
</p>
|
||||
|
||||
同一套引擎、同一个 int4 容器——硬件只会改变专家的存放位置。
|
||||
[完整 benchmark 表格](docs/benchmarks.md)中的重点如下：
|
||||
|
||||
- **6× RTX 5090，全部常驻：**解码 5.8–6.8 tok/s，TTFT 约 13 秒
|
||||
（[实验记录](docs/experiments/glm52-6x5090-2026-07-12.md)）；
|
||||
- **128 GB、仅使用 CPU 的台式机：**热缓存后约 1.8 tok/s
|
||||
（[#200](https://github.com/JustVugg/colibri/issues/200)）；
|
||||
- **单张 RTX 5070 Ti 的笔记本级主机：**通过 GPU 常驻管线达到 1.07 tok/s
|
||||
（[#273](https://github.com/JustVugg/colibri/issues/273)）；
|
||||
- **25 GB 开发机：**冷启动 0.05–0.1 tok/s——这是项目起步时已证实的下限，
|
||||
也仍是诚实的基准。
|
||||
|
||||
质量来自测量，而非假设：int4 容器的量化损失，以及 scale granularity／rotation
|
||||
消融实验，收录于 [docs/benchmarks.md](docs/benchmarks.md#quality-benchmark)、
|
||||
[#108](https://github.com/JustVugg/colibri/issues/108) 与
|
||||
[#81](https://github.com/JustVugg/colibri/issues/81)。
|
||||
|
||||
## 开始使用
|
||||
|
||||
### 1. 获取模型
|
||||
|
||||
Hugging Face 上已有预转换的 **GLM-5.2 int4** 容器——请务必使用
|
||||
**含 int8 MTP head 的版本**：
|
||||
|
||||
**https://huggingface.co/mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp**
|
||||
|
||||
> ⚠️ 原始镜像使用 int4 MTP head → 草稿接受率为 0%
|
||||
>（[#8](https://github.com/JustVugg/colibri/issues/8)）。请检查你的版本：
|
||||
> `ls -l <model>/out-mtp-*`——正确的 int8 大小为 `3527131672 / 5366238584 / 1065950496`。
|
||||
|
||||
你也可以自行从 FP8 源转换——只需一条可断点续传的命令，且任何时候都不需要
|
||||
在磁盘上同时存放完整的 756 GB：
|
||||
|
||||
```bash
|
||||
cd c && ./setup.sh # 检查 gcc/OpenMP、构建并运行自测
|
||||
./coli convert --model /nvme/glm52_i4 # 逐 shard 下载并转换（仅此一次需要 python）
|
||||
```
|
||||
|
||||
### 2. 运行
|
||||
|
||||
```bash
|
||||
COLI_MODEL=/nvme/glm52_i4 ./coli chat # 自动检测 RAM 预算、缓存与 MTP
|
||||
COLI_MODEL=/nvme/glm52_i4 ./coli plan # 查看规划的 VRAM／RAM／磁盘配置
|
||||
COLI_MODEL=/nvme/glm52_i4 ./coli doctor # 只读就绪检查
|
||||
./coli web --model /nvme/glm52_i4 # 在同一端口提供 API 与网页仪表盘
|
||||
./coli serve --model /nvme/glm52_i4 # 仅提供 OpenAI 兼容 API
|
||||
```
|
||||
|
||||
引擎运行时是纯 C——python 只供一次性转换工具与可选的 API gateway 使用。
|
||||
|
||||
### 3. 深入了解
|
||||
|
||||
| 主题 | 文档 |
|
||||
|---|---|
|
||||
| Benchmark、社区实测数据、质量测量 | [docs/benchmarks.md](docs/benchmarks.md) |
|
||||
| 调优选项、策略、学习型缓存、预取 | [docs/tuning.md](docs/tuning.md) |
|
||||
| Windows 11 原生构建（含 CUDA DLL） | [docs/windows.md](docs/windows.md) |
|
||||
| CUDA 后端、VRAM 专家层级、全部常驻 | [docs/cuda.md](docs/cuda.md) |
|
||||
| Apple Silicon Metal 后端 | [docs/metal.md](docs/metal.md) |
|
||||
| OpenAI 兼容 API、KV slots、网页仪表盘 | [docs/api.md](docs/api.md) |
|
||||
| 语法强制草稿（结构化输出） | [docs/grammar-draft.md](docs/grammar-draft.md) |
|
||||
| 环境变量完整清单 | [docs/ENVIRONMENT.md](docs/ENVIRONMENT.md) |
|
||||
|
||||
## 支持项目
|
||||
|
||||
colibrì 最初由一人使用 12 核心、25 GB RAM 的笔记本开发；
|
||||
如今它的数据来自社区中各种真实机器。如果这个项目对你有用：
|
||||
|
||||
- ⭐ 为仓库加星并分享；
|
||||
- 🐛 以 issue 提交你的硬件 benchmark 数据——实测数据比任何其他事都更能推动项目；
|
||||
- 💬 若想赞助开发或捐赠硬件，请通过 GitHub issues 联系。
|
||||
|
||||
## 仓库结构
|
||||
|
||||
```
|
||||
Makefile 根目录构建／检查入口
|
||||
c/
|
||||
├── colibri.c 引擎主文件
|
||||
├── quant.h 量化 matmul 内核（SIMD 多架构）
|
||||
├── sample.h 采样与 stop-set 管理
|
||||
├── kv_persist.h .coli_kv 磁盘持久化
|
||||
├── telemetry.h 仪表盘协议、统计与用量持久化
|
||||
├── st.h, tok.h, json.h 运行时头文件
|
||||
├── backend_cuda.* 可选的 CUDA 层级
|
||||
├── Makefile 构建与本地检查
|
||||
├── coli 用户界面 CLI
|
||||
├── openai_server.py OpenAI 兼容 HTTP gateway
|
||||
├── setup.sh 一条命令完成本地设置
|
||||
├── tools/ 离线转换、fixtures 与 benchmarks
|
||||
├── scripts/ 长时间转换辅助工具
|
||||
└── tests/ 零依赖的 C 与 Python 测试
|
||||
web/ 浏览器 UI（纯 OpenAI API client）
|
||||
desktop/ 封装网页 UI 的 Tauri v2 桌面 shell
|
||||
docs/ 参考文档、实验与媒体文件
|
||||
```
|
||||
|
||||
运行时路径刻意保持扁平、易读：`colibri.c` 加上若干头文件。
|
||||
在仓库根目录执行 `make`、`make check` 与 `make clean`，
|
||||
都会转发给引擎的 Makefile。
|
||||
|
||||
## 为什么叫"colibrì"
|
||||
|
||||
蜂鸟只有几克重，能在原地悬停，并在一天内造访上千朵花。
|
||||
这套引擎只用蜂鸟般的配给，就能让 744B 参数的巨人运转：
|
||||
25 GB RAM、十二个 CPU 核心，以及对磁盘的大量耐心。
|
||||
|
||||
## 许可证
|
||||
|
||||
Apache 2.0。GLM-5.2 权重由 Z.ai 以 MIT 许可发布。
|
||||
+8
-4
@@ -3,7 +3,7 @@
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="README.md">English</a> · 繁體中文
|
||||
<a href="README.md">English</a> · <a href="README.zh-CN.md">简体中文</a> · 繁體中文 · <a href="README.it.md">Italiano</a>
|
||||
</p>
|
||||
|
||||
**小巧引擎，龐大模型。**只要約 25 GB 記憶體，就能在消費級電腦上執行 **GLM-5.2（744B 參數的 MoE）**——以零相依套件的純 C 實作，從硬碟串流載入專家。
|
||||
@@ -60,7 +60,7 @@ VRAM／RAM／硬碟層級長條，以及角落的即時迷你大腦。</em></p>
|
||||
**存放在硬碟**（約 370 GB），並**隨需串流載入**，搭配逐層 LRU 快取、
|
||||
會學習的熱門專家固定儲存區，以及選用的 VRAM 層級。
|
||||
|
||||
引擎由單一 C 檔（`c/glm.c`）與少量標頭檔組成。不需要 BLAS，
|
||||
引擎由主 C 檔（`c/colibri.c`）與多個標頭檔模組組成。不需要 BLAS，
|
||||
執行階段不需要 Python，也不需要 GPU。
|
||||
|
||||
## 運作方式
|
||||
@@ -203,7 +203,11 @@ colibrì 最初是由一人使用 12 核心、25 GB RAM 的筆電開發；
|
||||
```
|
||||
Makefile 根目錄建置／檢查入口
|
||||
c/
|
||||
├── glm.c 單檔 GLM 引擎
|
||||
├── colibri.c GLM 引擎主檔
|
||||
├── quant.h 量化 matmul kernel
|
||||
├── sample.h 取樣與 stop-set
|
||||
├── kv_persist.h .coli_kv 磁碟持久化
|
||||
├── telemetry.h 儀表板協定、統計
|
||||
├── st.h, tok.h, json.h 執行階段標頭檔
|
||||
├── backend_cuda.* 選用的 CUDA 層級
|
||||
├── Makefile 建置與本機檢查
|
||||
@@ -218,7 +222,7 @@ desktop/ 包裝網頁 UI 的 Tauri v2 桌面 shell
|
||||
docs/ 參考文件、實驗與媒體檔
|
||||
```
|
||||
|
||||
執行階段路徑刻意維持扁平、易讀：`glm.c` 加上少量標頭檔。
|
||||
執行階段路徑刻意維持扁平、易讀：`colibri.c` 加上模組化標頭檔。
|
||||
在儲存庫根目錄執行 `make`、`make check` 與 `make clean`，
|
||||
都會轉交給引擎的 Makefile。
|
||||
|
||||
|
||||
+28
-27
@@ -56,7 +56,7 @@ OMPL =
|
||||
endif
|
||||
CFLAGS = -O3 $(OMPC) -Wall -Wextra -Wno-unused-parameter -Wno-misleading-indentation -Wno-unused-function
|
||||
# Opt-in: ARCH=native appends -mcpu=native (arm64 clang uses -mcpu, not -march),
|
||||
# which unlocks the i8mm SMMLA int8/int4 dot kernels in glm.c. ARCH unset ->
|
||||
# which unlocks the i8mm SMMLA int8/int4 dot kernels in colibri.c. ARCH unset ->
|
||||
# no -mcpu, default build byte-identical. Apple clang knows apple-m4 / native.
|
||||
ifneq ($(ARCH),)
|
||||
CFLAGS += -mcpu=$(ARCH)
|
||||
@@ -70,7 +70,7 @@ else ifneq ($(IS_WIN),)
|
||||
# ARCH default = x86-64-v3 (portable binary with AVX2). For max speed on THIS
|
||||
# machine use ARCH=native: on AVX-VNNI CPUs (Intel Alder Lake+, Meteor Lake+)
|
||||
# it also unlocks the 128-bit VPDPBUSD int8/int4 dot kernel (dot_i8i8/dot_i4i8),
|
||||
# which the x86-64-v3 baseline does not define. The #ifdef guards in glm.c mean
|
||||
# which the x86-64-v3 baseline does not define. The #ifdef guards in colibri.c mean
|
||||
# a v3 build simply compiles out the VNNI path - safe on any x86-64.
|
||||
CC = gcc
|
||||
ARCH ?= x86-64-v3
|
||||
@@ -207,17 +207,18 @@ LDFLAGS += -framework Metal -framework Foundation -lc++
|
||||
METAL_OBJ = backend_metal.o
|
||||
endif
|
||||
|
||||
all: glm$(EXE)
|
||||
all: colibri$(EXE)
|
||||
|
||||
# phony 'glm' → 'glm.exe' on Windows (so 'make glm' and 'coli build' work on every platform)
|
||||
glm: glm$(EXE)
|
||||
# phony targets — 'glm' kept for backward compatibility
|
||||
colibri: colibri$(EXE)
|
||||
glm: colibri$(EXE)
|
||||
|
||||
# Config stamp: make only tracks file timestamps, not flag changes. Without this,
|
||||
# `make glm.exe CUDA_DLL=1` after a prior CPU-only build reports "up to date" and
|
||||
# silently keeps the CPU-only binary (no CUDA loader) — a build that looks like it
|
||||
# worked but isn't. We record the build-affecting flags in .build-config and rewrite
|
||||
# it ONLY when they change (evaluated here at parse time, so the file's timestamp
|
||||
# moves exactly when the config moves). glm.exe and the CUDA/loader objects depend
|
||||
# `make colibri.exe CUDA_DLL=1` after a prior CPU-only build reports "up to date"
|
||||
# and silently keeps the CPU-only binary (no CUDA loader) — a build that looks like
|
||||
# it worked but isn't. We record the build-affecting flags in .build-config and
|
||||
# rewrite it ONLY when they change (evaluated here at parse time, so the file's
|
||||
# timestamp moves exactly when the config moves). The binary and CUDA/loader objects depend
|
||||
# on it, so they relink on a config change and stay put otherwise. (#306)
|
||||
BUILD_CONFIG := $(CC)|$(CFLAGS)|$(LDFLAGS)|CUDA=$(CUDA)|CUDA_DLL=$(CUDA_DLL)|ARCH=$(ARCH)|CUDA_ARCH=$(CUDA_ARCH)|METAL=$(METAL)
|
||||
BUILD_CONFIG_OLD := $(shell cat .build-config 2>/dev/null)
|
||||
@@ -226,8 +227,8 @@ $(shell printf '%s' '$(BUILD_CONFIG)' > .build-config)
|
||||
endif
|
||||
.build-config: ;
|
||||
|
||||
glm$(EXE): glm.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h $(CUDA_OBJ) $(METAL_OBJ) .build-config
|
||||
$(CC) $(CFLAGS) glm.c $(CUDA_OBJ) $(METAL_OBJ) -o glm$(EXE) $(LDFLAGS)
|
||||
colibri$(EXE): colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h quant.h sample.h kv_persist.h telemetry.h $(CUDA_OBJ) $(METAL_OBJ) .build-config
|
||||
$(CC) $(CFLAGS) colibri.c $(CUDA_OBJ) $(METAL_OBJ) -o colibri$(EXE) $(LDFLAGS)
|
||||
|
||||
# Windows runtime loader object: resolves coli_cuda_* from coli_cuda.dll.
|
||||
backend_loader.o: backend_loader.c backend_cuda.h compat.h .build-config
|
||||
@@ -285,7 +286,7 @@ PORTABLE_ARCH = native
|
||||
endif
|
||||
|
||||
portable:
|
||||
$(MAKE) glm$(EXE) ARCH=$(PORTABLE_ARCH)
|
||||
$(MAKE) colibri$(EXE) ARCH=$(PORTABLE_ARCH)
|
||||
|
||||
iobench$(EXE): iobench.c compat.h
|
||||
$(CC) $(CFLAGS) iobench.c -o iobench$(EXE) $(LDFLAGS)
|
||||
@@ -311,30 +312,30 @@ tests/test_schema_gbnf$(EXE): tests/test_schema_gbnf.c schema_gbnf.h grammar.h j
|
||||
tests/test_decode_batch$(EXE): tests/test_decode_batch.c decode_batch.h
|
||||
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
|
||||
|
||||
tests/test_idot$(EXE): tests/test_idot.c glm.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h
|
||||
tests/test_idot$(EXE): tests/test_idot.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h
|
||||
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
|
||||
|
||||
tests/test_i4_grouped$(EXE): tests/test_i4_grouped.c glm.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h
|
||||
tests/test_i4_grouped$(EXE): tests/test_i4_grouped.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h
|
||||
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
|
||||
|
||||
tests/test_stops$(EXE): tests/test_stops.c glm.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h
|
||||
tests/test_stops$(EXE): tests/test_stops.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h
|
||||
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
|
||||
|
||||
tests/test_topp$(EXE): tests/test_topp.c glm.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h
|
||||
tests/test_topp$(EXE): tests/test_topp.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h
|
||||
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
|
||||
|
||||
# bench_topp is a microbenchmark (old qsort vs new heap partial-select, #335), NOT a test
|
||||
# gate -- intentionally absent from TEST_BINS. Build on demand: make tests/bench_topp
|
||||
tests/bench_topp$(EXE): tests/bench_topp.c glm.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h
|
||||
tests/bench_topp$(EXE): tests/bench_topp.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h
|
||||
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
|
||||
|
||||
tests/test_sample_nan$(EXE): tests/test_sample_nan.c glm.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h
|
||||
tests/test_sample_nan$(EXE): tests/test_sample_nan.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h
|
||||
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
|
||||
|
||||
tests/test_kv_alloc$(EXE): tests/test_kv_alloc.c glm.c st.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h
|
||||
tests/test_kv_alloc$(EXE): tests/test_kv_alloc.c colibri.c st.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h
|
||||
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
|
||||
|
||||
tests/test_logit_nan$(EXE): tests/test_logit_nan.c glm.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h
|
||||
tests/test_logit_nan$(EXE): tests/test_logit_nan.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h
|
||||
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
|
||||
|
||||
tests/test_i4_acc512$(EXE): tests/test_i4_acc512.c
|
||||
@@ -343,15 +344,15 @@ tests/test_i4_acc512$(EXE): tests/test_i4_acc512.c
|
||||
tests/test_compat_direct$(EXE): tests/test_compat_direct.c compat.h
|
||||
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
|
||||
|
||||
tests/test_dsa_select$(EXE): tests/test_dsa_select.c glm.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h
|
||||
tests/test_dsa_select$(EXE): tests/test_dsa_select.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h
|
||||
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
|
||||
|
||||
# bench_dsa_select is a microbenchmark (old qsort vs new quickselect partial-select, #356),
|
||||
# NOT a test gate -- intentionally absent from TEST_BINS. Build on demand: make tests/bench_dsa_select
|
||||
tests/bench_dsa_select$(EXE): tests/bench_dsa_select.c glm.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h
|
||||
tests/bench_dsa_select$(EXE): tests/bench_dsa_select.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h
|
||||
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
|
||||
|
||||
tests/test_uring$(EXE): tests/test_uring.c glm.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h
|
||||
tests/test_uring$(EXE): tests/test_uring.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h
|
||||
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
|
||||
|
||||
test-c: $(TEST_BINS)
|
||||
@@ -368,12 +369,12 @@ check:
|
||||
$(MAKE) portable
|
||||
$(MAKE) test
|
||||
|
||||
install: glm$(EXE) olmoe$(EXE)
|
||||
install: colibri$(EXE) olmoe$(EXE)
|
||||
$(INSTALL) -d $(DESTDIR)$(BINDIR)
|
||||
$(INSTALL) -d $(DESTDIR)$(LIBEXECDIR)
|
||||
$(INSTALL) -d $(DESTDIR)$(LIBEXECDIR)/tools
|
||||
$(INSTALL) -m 755 coli $(DESTDIR)$(BINDIR)/coli
|
||||
$(INSTALL) -m 755 glm$(EXE) $(DESTDIR)$(LIBEXECDIR)/glm$(EXE)
|
||||
$(INSTALL) -m 755 colibri$(EXE) $(DESTDIR)$(LIBEXECDIR)/colibri$(EXE)
|
||||
$(INSTALL) -m 755 olmoe$(EXE) $(DESTDIR)$(LIBEXECDIR)/olmoe$(EXE)
|
||||
$(INSTALL) -m 644 resource_plan.py doctor.py openai_server.py $(DESTDIR)$(LIBEXECDIR)/
|
||||
$(INSTALL) -m 644 tools/*.py $(DESTDIR)$(LIBEXECDIR)/tools/
|
||||
@@ -387,4 +388,4 @@ clean:
|
||||
|
||||
bench: iobench$(EXE)
|
||||
@if [ -n "$(ARGS)" ]; then ./iobench$(EXE) $(ARGS); else echo "built iobench$(EXE) — run: ./iobench$(EXE) <file> <MB> <iters> <threads> <direct 0|1>"; fi
|
||||
.PHONY: all glm cuda-test cuda-bench cuda-dll portable test-c test-python test check clean install uninstall bench
|
||||
.PHONY: all colibri glm cuda-test cuda-bench cuda-dll portable test-c test-python test check clean install uninstall bench
|
||||
|
||||
@@ -54,16 +54,20 @@ from version import __version__ as _version
|
||||
# guess is right (e.g. a custom packaging layout).
|
||||
_EXE = ".exe" if sys.platform == "win32" else ""
|
||||
_LIBEXEC = os.path.join(os.path.dirname(HERE), "libexec", "colibri")
|
||||
_here_colibri = os.path.join(HERE, "colibri" + _EXE)
|
||||
_here_glm = os.path.join(HERE, "glm" + _EXE)
|
||||
|
||||
if os.environ.get("COLI_ENGINE"):
|
||||
GLM = os.environ["COLI_ENGINE"]
|
||||
TOOLS = os.path.join(os.path.dirname(GLM), "tools")
|
||||
elif os.path.exists(_here_colibri):
|
||||
GLM = _here_colibri
|
||||
TOOLS = os.path.join(HERE, "tools")
|
||||
elif os.path.exists(_here_glm):
|
||||
GLM = _here_glm
|
||||
TOOLS = os.path.join(HERE, "tools")
|
||||
else:
|
||||
GLM = os.path.join(_LIBEXEC, "glm" + _EXE)
|
||||
GLM = os.path.join(_LIBEXEC, "colibri" + _EXE)
|
||||
TOOLS = os.path.join(_LIBEXEC, "tools")
|
||||
sys.path.insert(0, _LIBEXEC) # so `import resource_plan`, `doctor`, `openai_server` still resolve
|
||||
|
||||
@@ -241,13 +245,13 @@ def env_for(a):
|
||||
e["COLI_CUDA"]="0"; e.pop("CUDA_EXPERT_GB",None); e.pop("CUDA_DENSE",None)
|
||||
else:
|
||||
if not cuda_binary():
|
||||
sys.exit(f"{C.yel}--gpu needs the CUDA build:{C.r} make glm CUDA=1 (this binary is CPU-only)")
|
||||
sys.exit(f"{C.yel}--gpu needs the CUDA build:{C.r} make colibri CUDA=1 (this binary is CPU-only)")
|
||||
e["COLI_CUDA"]="1"
|
||||
if a.gpu!="auto": e["COLI_GPUS"]=a.gpu
|
||||
e.setdefault("CUDA_DENSE","1")
|
||||
if a.vram and a.gpu!="none":
|
||||
if not cuda_binary():
|
||||
sys.exit(f"{C.yel}--vram needs the CUDA build:{C.r} make glm CUDA=1 (this binary is CPU-only)")
|
||||
sys.exit(f"{C.yel}--vram needs the CUDA build:{C.r} make colibri CUDA=1 (this binary is CPU-only)")
|
||||
e["COLI_CUDA"]="1"; e["CUDA_EXPERT_GB"]=str(a.vram)
|
||||
return e
|
||||
|
||||
@@ -421,8 +425,8 @@ def cmd_build(a):
|
||||
banner("build")
|
||||
if not os.path.exists(os.path.join(HERE, "Makefile")):
|
||||
sys.exit(f"{C.yel}coli build{C.r} only works from a source checkout (this is an installed copy).\n"
|
||||
f" Clone https://github.com/JustVugg/colibri and run ./setup.sh, or make -C c glm.")
|
||||
sys.exit(subprocess.call(["make","-C",HERE,"glm"]))
|
||||
f" Clone https://github.com/JustVugg/colibri and run ./setup.sh, or make -C c colibri.")
|
||||
sys.exit(subprocess.call(["make","-C",HERE,"colibri"]))
|
||||
|
||||
def cmd_info(a):
|
||||
banner("info")
|
||||
@@ -802,7 +806,7 @@ def cmd_stop(a):
|
||||
if "coli" in cmd and " serve" in cmd and pid!=os.getpid():
|
||||
if not any(p==pid for p,_ in targets): targets.append((pid,"coli serve (cmdline)"))
|
||||
comm=open(f"/proc/{pd}/comm").read().strip()
|
||||
if comm in ("glm","exe","olmoe"):
|
||||
if comm in ("colibri","glm","exe","olmoe"):
|
||||
env=open(f"/proc/{pd}/environ","rb").read().replace(b"\0",b"\n").decode("utf-8","replace")
|
||||
if "SERVE=1" in env: targets.append((pid,f"engine `{comm}` (SERVE=1)"))
|
||||
except (OSError,PermissionError): continue
|
||||
|
||||
+24
-1220
File diff suppressed because it is too large
Load Diff
+121
@@ -0,0 +1,121 @@
|
||||
/* kv_persist.h — .coli_kv on-disk KV cache persistence.
|
||||
* Conversations reopen warm across engine restarts: the compressed MLA KV-cache
|
||||
* is appended incrementally after every turn, crash-safe (nrec written last).
|
||||
* Include after Model/KVState/Cfg are defined; requires now_s() and g_draft. */
|
||||
#ifndef KV_PERSIST_H
|
||||
#define KV_PERSIST_H
|
||||
|
||||
static int g_kvsave=1;
|
||||
#define KV_MAGIC "COLIKV1\0"
|
||||
|
||||
static void kv_hdr(Model *m, int32_t *h, int nrec){
|
||||
Cfg *c=&m->c; int nic=0;
|
||||
for(int i=0;i<c->n_layers;i++) if(m->Ic && m->Ic[i]) nic++;
|
||||
h[0]=c->n_layers; h[1]=c->kv_lora; h[2]=c->qk_rope;
|
||||
h[3]=m->has_dsa?c->index_hd:0; h[4]=nic; h[5]=c->vocab; h[6]=nrec; h[7]=0;
|
||||
}
|
||||
|
||||
static int64_t kv_rec_bytes(Model *m){
|
||||
Cfg *c=&m->c;
|
||||
int64_t rec = 4 + (int64_t)c->n_layers*(c->kv_lora+c->qk_rope)*4;
|
||||
if(m->has_dsa) for(int i=0;i<c->n_layers;i++) if(m->Ic[i]) rec+=(int64_t)c->index_hd*4;
|
||||
return rec;
|
||||
}
|
||||
|
||||
static int kv_disk_open(Model *m){
|
||||
KVState *k=m->kv;
|
||||
if(k->disk_fp) return 1;
|
||||
k->disk_fp=fopen(k->disk_path,"r+b");
|
||||
if(!k->disk_fp){
|
||||
k->disk_fp=fopen(k->disk_path,"wb");
|
||||
if(!k->disk_fp) return 0;
|
||||
int32_t h[8]; kv_hdr(m,h,0);
|
||||
fwrite(KV_MAGIC,1,8,k->disk_fp); fwrite(h,4,8,k->disk_fp);
|
||||
fflush(k->disk_fp);
|
||||
fclose(k->disk_fp);
|
||||
k->disk_fp=fopen(k->disk_path,"r+b");
|
||||
if(!k->disk_fp) return 0;
|
||||
}
|
||||
return 1;
|
||||
}
|
||||
|
||||
static void kv_disk_truncate(Model *m, int nrec){
|
||||
if(!g_kvsave) return;
|
||||
KVState *k=m->kv;
|
||||
if(k->disk_fp){ fclose(k->disk_fp); k->disk_fp=NULL; }
|
||||
FILE *f=fopen(k->disk_path,"r+b");
|
||||
if(!f){ k->disk_nrec=0; return; }
|
||||
k->disk_nrec=nrec;
|
||||
int32_t nr=nrec; fseek(f,8+6*4,SEEK_SET); fwrite(&nr,4,1,f);
|
||||
fflush(f); fclose(f);
|
||||
}
|
||||
|
||||
static void kv_disk_reset(Model *m){ kv_disk_truncate(m,0); }
|
||||
|
||||
static void kv_disk_append(Model *m, const int *hist, int len){
|
||||
KVState *k=m->kv;
|
||||
if(!g_kvsave || len<=k->disk_nrec) return;
|
||||
Cfg *c=&m->c;
|
||||
if(!kv_disk_open(m)) return;
|
||||
FILE *f=k->disk_fp;
|
||||
int64_t rec = kv_rec_bytes(m);
|
||||
if(rec > k->disk_buf_cap){
|
||||
uint8_t *nb=realloc(k->disk_buf, rec);
|
||||
if(!nb) return;
|
||||
k->disk_buf=nb; k->disk_buf_cap=rec;
|
||||
}
|
||||
fseek(f, 8+8*4 + (int64_t)k->disk_nrec*rec, SEEK_SET);
|
||||
for(int p=k->disk_nrec;p<len;p++){
|
||||
uint8_t *b=k->disk_buf;
|
||||
*(int32_t*)b = hist[p]; b+=4;
|
||||
for(int i=0;i<c->n_layers;i++){
|
||||
memcpy(b, m->Lc[i]+(int64_t)p*c->kv_lora, (size_t)c->kv_lora*4); b+=c->kv_lora*4;
|
||||
memcpy(b, m->Rc[i]+(int64_t)p*c->qk_rope,(size_t)c->qk_rope*4); b+=c->qk_rope*4;
|
||||
}
|
||||
if(m->has_dsa) for(int i=0;i<c->n_layers;i++) if(m->Ic[i]){
|
||||
memcpy(b, m->Ic[i]+(int64_t)p*c->index_hd, (size_t)c->index_hd*4); b+=c->index_hd*4;
|
||||
}
|
||||
fwrite(k->disk_buf, 1, (size_t)rec, f);
|
||||
}
|
||||
fflush(f);
|
||||
int32_t nr=len; fseek(f,8+6*4,SEEK_SET); fwrite(&nr,4,1,f);
|
||||
fflush(f);
|
||||
k->disk_nrec=len;
|
||||
}
|
||||
|
||||
static int kv_disk_load(Model *m, int *hist, int maxctx){
|
||||
if(!g_kvsave) return 0;
|
||||
KVState *k=m->kv;
|
||||
Cfg *c=&m->c;
|
||||
FILE *f=fopen(k->disk_path,"rb"); if(!f) return 0;
|
||||
char mg[8]; int32_t h[8], w[8]; kv_hdr(m,w,0);
|
||||
if(fread(mg,1,8,f)!=8 || memcmp(mg,KV_MAGIC,8) || fread(h,4,8,f)!=8 ||
|
||||
h[0]!=w[0]||h[1]!=w[1]||h[2]!=w[2]||h[3]!=w[3]||h[4]!=w[4]||h[5]!=w[5]){
|
||||
fprintf(stderr,"[KV] ignoring .coli_kv from a different model or version\n"); fclose(f); return 0; }
|
||||
int nrec=h[6];
|
||||
if(nrec<1){ fclose(f); return 0; }
|
||||
if(nrec>=maxctx-8-g_draft){
|
||||
fprintf(stderr,"[KV] saved conversation (%d tokens) exceeds the context: starting over\n",nrec);
|
||||
fclose(f); return 0; }
|
||||
double t0=now_s();
|
||||
for(int p=0;p<nrec;p++){
|
||||
int32_t tk; if(fread(&tk,4,1,f)!=1){ nrec=p; break; } hist[p]=tk;
|
||||
for(int i=0;i<c->n_layers;i++){
|
||||
if(fread(m->Lc[i]+(int64_t)p*c->kv_lora, 4, c->kv_lora, f)!=(size_t)c->kv_lora ||
|
||||
fread(m->Rc[i]+(int64_t)p*c->qk_rope, 4, c->qk_rope, f)!=(size_t)c->qk_rope){ nrec=p; goto out; }
|
||||
}
|
||||
if(m->has_dsa) for(int i=0;i<c->n_layers;i++) if(m->Ic[i])
|
||||
if(fread(m->Ic[i]+(int64_t)p*c->index_hd, 4, c->index_hd, f)!=(size_t)c->index_hd){ nrec=p; goto out; }
|
||||
}
|
||||
out:
|
||||
fclose(f);
|
||||
if(nrec>0){
|
||||
if(m->has_mtp) m->kv_start[c->n_layers]=-1;
|
||||
fprintf(stderr,"[KV] resumed conversation from disk: %d tokens in %.1fs (no re-prefill)\n",
|
||||
nrec, now_s()-t0);
|
||||
}
|
||||
k->disk_nrec=nrec;
|
||||
return nrec;
|
||||
}
|
||||
|
||||
#endif /* KV_PERSIST_H */
|
||||
@@ -0,0 +1,672 @@
|
||||
/* quant.h — quantized matmul kernels (header-only, all functions static).
|
||||
* Multi-architecture SIMD: AVX2 / AVX-512 / AVX-VNNI / ARM NEON / NEON-SDOT /
|
||||
* NEON-i8mm / POWER VSX. Pure compute — no Model or QT dependency. */
|
||||
#ifndef COLI_QUANT_H
|
||||
#define COLI_QUANT_H
|
||||
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <math.h>
|
||||
#include <stdint.h>
|
||||
|
||||
#ifdef _OPENMP
|
||||
#include <omp.h>
|
||||
#endif
|
||||
|
||||
/* ---- SIMD includes -------------------------------------------------------- */
|
||||
#ifdef __AVX2__
|
||||
#include <immintrin.h>
|
||||
static inline float hsum256(__m256 v){
|
||||
__m128 lo=_mm256_castps256_ps128(v), hi=_mm256_extractf128_ps(v,1);
|
||||
lo=_mm_add_ps(lo,hi); __m128 sh=_mm_movehl_ps(lo,lo); lo=_mm_add_ps(lo,sh);
|
||||
sh=_mm_shuffle_ps(lo,lo,1); lo=_mm_add_ss(lo,sh); return _mm_cvtss_f32(lo);
|
||||
}
|
||||
static inline int hsum256_i32(__m256i v){
|
||||
__m128i lo=_mm256_castsi256_si128(v), hi=_mm256_extracti128_si256(v,1);
|
||||
lo=_mm_add_epi32(lo,hi); lo=_mm_hadd_epi32(lo,lo); lo=_mm_hadd_epi32(lo,lo);
|
||||
return _mm_cvtsi128_si32(lo);
|
||||
}
|
||||
#endif
|
||||
#if defined(__AVXVNNI__) && defined(__AVX2__)
|
||||
static inline int hsum128_i32(__m128i v){
|
||||
v=_mm_hadd_epi32(v,v); v=_mm_hadd_epi32(v,v); return _mm_cvtsi128_si32(v);
|
||||
}
|
||||
#endif
|
||||
#ifdef __ARM_NEON
|
||||
#include <arm_neon.h>
|
||||
#endif
|
||||
#ifdef __VSX__
|
||||
#include <altivec.h>
|
||||
#undef vector
|
||||
#undef pixel
|
||||
#undef bool
|
||||
#endif
|
||||
|
||||
/* ---- AVX-512 int4->float accumulator -------------------------------------- */
|
||||
#if defined(__AVX512F__) && defined(__AVX512BW__)
|
||||
static int g_i4_acc512=1;
|
||||
static inline float dot_i4f_avx512(const uint8_t *w,const float *x,int I){
|
||||
const __m128i m4=_mm_set1_epi8(0x0F); const __m512i b8=_mm512_set1_epi32(8);
|
||||
__m512 acc0=_mm512_setzero_ps(),acc1=_mm512_setzero_ps(); int i=0;
|
||||
for(;i+32<=I;i+=32){ __m128i by=_mm_loadu_si128((const __m128i*)(w+(i>>1)));
|
||||
__m128i lo=_mm_and_si128(by,m4),hi=_mm_and_si128(_mm_srli_epi16(by,4),m4);
|
||||
__m128i n0=_mm_unpacklo_epi8(lo,hi),n1=_mm_unpackhi_epi8(lo,hi);
|
||||
__m512 w0=_mm512_cvtepi32_ps(_mm512_sub_epi32(_mm512_cvtepu8_epi32(n0),b8));
|
||||
__m512 w1=_mm512_cvtepi32_ps(_mm512_sub_epi32(_mm512_cvtepu8_epi32(n1),b8));
|
||||
acc0=_mm512_fmadd_ps(_mm512_loadu_ps(x+i),w0,acc0);
|
||||
acc1=_mm512_fmadd_ps(_mm512_loadu_ps(x+i+16),w1,acc1);
|
||||
}
|
||||
return _mm512_reduce_add_ps(_mm512_add_ps(acc0,acc1));
|
||||
}
|
||||
static int i4_acc512_selftest(void){
|
||||
enum { N=224 }; uint8_t w[(N+1)/2]; float x[N];
|
||||
for(int i=0;i<N;i++){
|
||||
int q=((i*13+5)&15)-8;
|
||||
if(!(i&1)) w[i>>1]=(uint8_t)(q+8);
|
||||
else w[i>>1]|=(uint8_t)((q+8)<<4);
|
||||
x[i]=(float)(((i*29+7)%101)-50)/37.f;
|
||||
}
|
||||
for(int n=32;n<=N;n+=32){
|
||||
float ref=0; for(int i=0;i<n;i++) ref+=x[i]*(float)(((w[i>>1]>>((i&1)*4))&15)-8);
|
||||
float got=dot_i4f_avx512(w,x,n),tol=2e-5f*(1.f+fabsf(ref));
|
||||
if(fabsf(got-ref)>tol){ fprintf(stderr,"AVX512 i4 selftest n=%d: %.9g != %.9g\n",n,got,ref); return 0; }
|
||||
}
|
||||
return 1;
|
||||
}
|
||||
#endif
|
||||
|
||||
/* ---- y[S,O] = x[S,I] @ W^T, W[O,I] f32 ---------------------------------- */
|
||||
static void matmul(float *y, const float *x, const float *W, int S, int I, int O){
|
||||
#pragma omp parallel for schedule(static)
|
||||
for (int o=0;o<O;o++){ const float *w=W+(int64_t)o*I;
|
||||
for (int s=0;s<S;s++){ const float *xs=x+(int64_t)s*I; float a=0; for(int i=0;i<I;i++) a+=xs[i]*w[i]; y[(int64_t)s*O+o]=a; } }
|
||||
}
|
||||
|
||||
/* ---- y[S,O] = x[S,I] @ W^T, W int8 per-row + scale[O] ------------------- */
|
||||
static void matmul_q(float *y, const float *x, const int8_t *q, const float *scale, int S, int I, int O){
|
||||
#pragma omp parallel for schedule(static)
|
||||
for (int o=0;o<O;o++){ const int8_t *w=q+(int64_t)o*I; float sc=scale[o];
|
||||
for (int s=0;s<S;s++){ const float *xs=x+(int64_t)s*I; float a=0; int i=0;
|
||||
#ifdef __AVX2__
|
||||
__m256 acc=_mm256_setzero_ps();
|
||||
for(;i+8<=I;i+=8){ __m256i wi=_mm256_cvtepi8_epi32(_mm_loadl_epi64((const __m128i*)(w+i)));
|
||||
acc=_mm256_fmadd_ps(_mm256_loadu_ps(xs+i), _mm256_cvtepi32_ps(wi), acc); }
|
||||
a=hsum256(acc);
|
||||
#elif defined(__ARM_NEON)
|
||||
float32x4_t ac0=vdupq_n_f32(0), ac1=vdupq_n_f32(0);
|
||||
for(;i+8<=I;i+=8){ int16x8_t w16=vmovl_s8(vld1_s8(w+i));
|
||||
ac0=vfmaq_f32(ac0, vld1q_f32(xs+i), vcvtq_f32_s32(vmovl_s16(vget_low_s16(w16))));
|
||||
ac1=vfmaq_f32(ac1, vld1q_f32(xs+i+4), vcvtq_f32_s32(vmovl_s16(vget_high_s16(w16)))); }
|
||||
a=vaddvq_f32(vaddq_f32(ac0,ac1));
|
||||
#endif
|
||||
for(;i<I;i++) a+=xs[i]*(float)w[i]; y[(int64_t)s*O+o]=a*sc; } }
|
||||
}
|
||||
|
||||
/* ---- y[S,O] = x[S,I] @ W^T, W int4 packed (2/byte) + scale[O] ------------ */
|
||||
static void matmul_i4(float *y, const float *x, const uint8_t *q4, const float *scale, int S, int I, int O){
|
||||
int rb=(I+1)/2;
|
||||
#pragma omp parallel for schedule(static)
|
||||
for (int o=0;o<O;o++){ const uint8_t *w=q4+(int64_t)o*rb; float sc=scale[o];
|
||||
for (int s=0;s<S;s++){ const float *xs=x+(int64_t)s*I; float a=0; int i=0;
|
||||
#if defined(__AVX512F__) && defined(__AVX512BW__)
|
||||
if(g_i4_acc512){ a=dot_i4f_avx512(w,xs,I); i=I&~31; }
|
||||
else {
|
||||
#endif
|
||||
#ifdef __AVX2__
|
||||
const __m128i m4=_mm_set1_epi8(0x0F); const __m256i b8=_mm256_set1_epi32(8);
|
||||
__m256 acc=_mm256_setzero_ps();
|
||||
for(;i+16<=I;i+=16){ __m128i by=_mm_loadl_epi64((const __m128i*)(w+(i>>1)));
|
||||
__m128i lo=_mm_and_si128(by,m4), hi=_mm_and_si128(_mm_srli_epi16(by,4),m4);
|
||||
__m128i nib=_mm_unpacklo_epi8(lo,hi);
|
||||
__m256 w0=_mm256_cvtepi32_ps(_mm256_sub_epi32(_mm256_cvtepu8_epi32(nib),b8));
|
||||
__m256 w1=_mm256_cvtepi32_ps(_mm256_sub_epi32(_mm256_cvtepu8_epi32(_mm_srli_si128(nib,8)),b8));
|
||||
acc=_mm256_fmadd_ps(_mm256_loadu_ps(xs+i), w0, acc);
|
||||
acc=_mm256_fmadd_ps(_mm256_loadu_ps(xs+i+8), w1, acc); }
|
||||
a=hsum256(acc);
|
||||
#elif defined(__ARM_NEON)
|
||||
const uint8x8_t m4=vdup_n_u8(0x0F); const int8x8_t b8=vdup_n_s8(8);
|
||||
float32x4_t ac0=vdupq_n_f32(0), ac1=vdupq_n_f32(0);
|
||||
for(;i+16<=I;i+=16){ uint8x8_t by=vld1_u8(w+(i>>1));
|
||||
uint8x8x2_t z=vzip_u8(vand_u8(by,m4), vshr_n_u8(by,4));
|
||||
int16x8_t w0=vmovl_s8(vsub_s8(vreinterpret_s8_u8(z.val[0]),b8));
|
||||
int16x8_t w1=vmovl_s8(vsub_s8(vreinterpret_s8_u8(z.val[1]),b8));
|
||||
ac0=vfmaq_f32(ac0, vld1q_f32(xs+i), vcvtq_f32_s32(vmovl_s16(vget_low_s16(w0))));
|
||||
ac1=vfmaq_f32(ac1, vld1q_f32(xs+i+4), vcvtq_f32_s32(vmovl_s16(vget_high_s16(w0))));
|
||||
ac0=vfmaq_f32(ac0, vld1q_f32(xs+i+8), vcvtq_f32_s32(vmovl_s16(vget_low_s16(w1))));
|
||||
ac1=vfmaq_f32(ac1, vld1q_f32(xs+i+12), vcvtq_f32_s32(vmovl_s16(vget_high_s16(w1)))); }
|
||||
a=vaddvq_f32(vaddq_f32(ac0,ac1));
|
||||
#endif
|
||||
#if defined(__AVX512F__) && defined(__AVX512BW__)
|
||||
}
|
||||
#endif
|
||||
for(;i+1<I;i+=2){ uint8_t byte=w[i>>1]; int lo=(int)(byte&0xF)-8, hi=(int)(byte>>4)-8;
|
||||
a += xs[i]*(float)lo + xs[i+1]*(float)hi; }
|
||||
if(i<I){ uint8_t byte=w[i>>1]; int lo=(int)(byte&0xF)-8; a += xs[i]*(float)lo; }
|
||||
y[(int64_t)s*O+o]=a*sc; } }
|
||||
}
|
||||
|
||||
/* ---- y[S,O] = x[S,I] @ W^T, W int4 packed + per-GROUP scales (fmt=4) ----- */
|
||||
static void matmul_i4_grouped(float *y, const float *x, const uint8_t *q4, const float *scale,
|
||||
int S, int I, int O, int gs){
|
||||
int rb=(I+1)/2; int ng=(I+gs-1)/gs;
|
||||
#pragma omp parallel for schedule(static)
|
||||
for(int o=0;o<O;o++){
|
||||
const uint8_t *w=q4+(int64_t)o*rb;
|
||||
const float *scl=scale+(int64_t)o*ng;
|
||||
for(int s=0;s<S;s++){
|
||||
const float *xs=x+(int64_t)s*I; float a=0;
|
||||
for(int g=0; g*gs<I; g++){
|
||||
int base=g*gs; int glen=gs; if(base+glen>I) glen=I-base;
|
||||
float sc=scl[g];
|
||||
int i=base;
|
||||
#ifdef __AVX2__
|
||||
const __m128i m4=_mm_set1_epi8(0x0F); const __m256i b8=_mm256_set1_epi32(8);
|
||||
__m256 acc=_mm256_setzero_ps();
|
||||
for(; i+16<=base+glen; i+=16){ __m128i by=_mm_loadl_epi64((const __m128i*)(w+(i>>1)));
|
||||
__m128i lo=_mm_and_si128(by,m4),hi=_mm_and_si128(_mm_srli_epi16(by,4),m4);
|
||||
__m128i nib=_mm_unpacklo_epi8(lo,hi);
|
||||
__m256 w0=_mm256_cvtepi32_ps(_mm256_sub_epi32(_mm256_cvtepu8_epi32(nib),b8));
|
||||
__m256 w1=_mm256_cvtepi32_ps(_mm256_sub_epi32(_mm256_cvtepu8_epi32(_mm_srli_si128(nib,8)),b8));
|
||||
acc=_mm256_fmadd_ps(_mm256_loadu_ps(xs+i), w0, acc);
|
||||
acc=_mm256_fmadd_ps(_mm256_loadu_ps(xs+i+8), w1, acc); }
|
||||
a+=hsum256(acc)*sc;
|
||||
#endif
|
||||
for(; i<base+glen; i+=2){
|
||||
if(i+1<base+glen){ uint8_t byte=w[i>>1];
|
||||
a+=(xs[i]*(float)((int)(byte&0xF)-8)+xs[i+1]*(float)((int)(byte>>4)-8))*sc; }
|
||||
else { uint8_t byte=w[i>>1]; a+=xs[i]*(float)((int)(byte&0xF)-8)*sc; }
|
||||
}
|
||||
}
|
||||
y[(int64_t)s*O+o]=a;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/* ---- fused gate+up: one OMP dispatch for both matrices -------------------- */
|
||||
static void matmul_i4_pair(float *yg, float *yu, const float *x,
|
||||
const uint8_t *qg, const float *sg,
|
||||
const uint8_t *qu, const float *su, int I, int O){
|
||||
int rb=(I+1)/2;
|
||||
#pragma omp parallel for schedule(static)
|
||||
for(int z=0;z<2*O;z++){
|
||||
int o=z<O?z:z-O; const uint8_t *w=(z<O?qg:qu)+(int64_t)o*rb;
|
||||
float a=0; int i=0;
|
||||
#if defined(__AVX512F__) && defined(__AVX512BW__)
|
||||
if(g_i4_acc512){ a=dot_i4f_avx512(w,x,I); i=I&~31; }
|
||||
else {
|
||||
#endif
|
||||
#ifdef __AVX2__
|
||||
const __m128i m4=_mm_set1_epi8(0x0F); const __m256i b8=_mm256_set1_epi32(8);
|
||||
__m256 acc=_mm256_setzero_ps();
|
||||
for(;i+16<=I;i+=16){ __m128i by=_mm_loadl_epi64((const __m128i*)(w+(i>>1)));
|
||||
__m128i lo=_mm_and_si128(by,m4),hi=_mm_and_si128(_mm_srli_epi16(by,4),m4);
|
||||
__m128i nib=_mm_unpacklo_epi8(lo,hi);
|
||||
__m256 w0=_mm256_cvtepi32_ps(_mm256_sub_epi32(_mm256_cvtepu8_epi32(nib),b8));
|
||||
__m256 w1=_mm256_cvtepi32_ps(_mm256_sub_epi32(_mm256_cvtepu8_epi32(_mm_srli_si128(nib,8)),b8));
|
||||
acc=_mm256_fmadd_ps(_mm256_loadu_ps(x+i),w0,acc);
|
||||
acc=_mm256_fmadd_ps(_mm256_loadu_ps(x+i+8),w1,acc); }
|
||||
a=hsum256(acc);
|
||||
#elif defined(__ARM_NEON)
|
||||
const uint8x8_t m4=vdup_n_u8(0x0F); const int8x8_t b8=vdup_n_s8(8);
|
||||
float32x4_t ac0=vdupq_n_f32(0),ac1=vdupq_n_f32(0);
|
||||
for(;i+16<=I;i+=16){ uint8x8_t by=vld1_u8(w+(i>>1));
|
||||
uint8x8x2_t n=vzip_u8(vand_u8(by,m4),vshr_n_u8(by,4));
|
||||
int16x8_t w0=vmovl_s8(vsub_s8(vreinterpret_s8_u8(n.val[0]),b8));
|
||||
int16x8_t w1=vmovl_s8(vsub_s8(vreinterpret_s8_u8(n.val[1]),b8));
|
||||
ac0=vfmaq_f32(ac0,vld1q_f32(x+i),vcvtq_f32_s32(vmovl_s16(vget_low_s16(w0))));
|
||||
ac1=vfmaq_f32(ac1,vld1q_f32(x+i+4),vcvtq_f32_s32(vmovl_s16(vget_high_s16(w0))));
|
||||
ac0=vfmaq_f32(ac0,vld1q_f32(x+i+8),vcvtq_f32_s32(vmovl_s16(vget_low_s16(w1))));
|
||||
ac1=vfmaq_f32(ac1,vld1q_f32(x+i+12),vcvtq_f32_s32(vmovl_s16(vget_high_s16(w1)))); }
|
||||
a=vaddvq_f32(vaddq_f32(ac0,ac1));
|
||||
#endif
|
||||
#if defined(__AVX512F__) && defined(__AVX512BW__)
|
||||
}
|
||||
#endif
|
||||
for(;i+1<I;i+=2){ uint8_t b=w[i>>1]; a+=x[i]*(float)((b&15)-8)+x[i+1]*(float)((b>>4)-8); }
|
||||
if(i<I) a+=x[i]*(float)((w[i>>1]&15)-8);
|
||||
(z<O?yg:yu)[o]=a*(z<O?sg:su)[o];
|
||||
}
|
||||
}
|
||||
|
||||
/* ---- y[S,O] = x[S,I] @ W^T, W int2 packed (4/byte) + scale[O] ------------ */
|
||||
static void matmul_i2(float *y, const float *x, const uint8_t *q2, const float *scale, int S, int I, int O){
|
||||
int rb=(I+3)/4;
|
||||
#pragma omp parallel for schedule(static)
|
||||
for (int o=0;o<O;o++){ const uint8_t *w=q2+(int64_t)o*rb; float sc=scale[o];
|
||||
for (int s=0;s<S;s++){ const float *xs=x+(int64_t)s*I; float a=0; int i=0;
|
||||
#ifdef __AVX2__
|
||||
const __m128i m2=_mm_set1_epi8(0x03); const __m256i b2=_mm256_set1_epi32(2);
|
||||
__m256 acc=_mm256_setzero_ps();
|
||||
for(;i+16<=I;i+=16){ __m128i by=_mm_cvtsi32_si128(*(const int*)(w+(i>>2)));
|
||||
__m128i p0=_mm_and_si128(by,m2), p1=_mm_and_si128(_mm_srli_epi16(by,2),m2);
|
||||
__m128i p2=_mm_and_si128(_mm_srli_epi16(by,4),m2), p3=_mm_and_si128(_mm_srli_epi16(by,6),m2);
|
||||
__m128i lo=_mm_unpacklo_epi8(p0,p1), hi=_mm_unpacklo_epi8(p2,p3);
|
||||
__m128i nib=_mm_unpacklo_epi16(lo,hi);
|
||||
__m256 w0=_mm256_cvtepi32_ps(_mm256_sub_epi32(_mm256_cvtepu8_epi32(nib),b2));
|
||||
__m256 w1=_mm256_cvtepi32_ps(_mm256_sub_epi32(_mm256_cvtepu8_epi32(_mm_srli_si128(nib,8)),b2));
|
||||
acc=_mm256_fmadd_ps(_mm256_loadu_ps(xs+i), w0, acc);
|
||||
acc=_mm256_fmadd_ps(_mm256_loadu_ps(xs+i+8), w1, acc); }
|
||||
a=hsum256(acc);
|
||||
#elif defined(__ARM_NEON)
|
||||
const uint8x8_t m2v=vdup_n_u8(3); const int8x8_t b2v=vdup_n_s8(2);
|
||||
float32x4_t ac0=vdupq_n_f32(0), ac1=vdupq_n_f32(0);
|
||||
for(;i+16<=I;i+=16){ uint32_t wd; memcpy(&wd, w+(i>>2), 4);
|
||||
uint8x8_t by=vreinterpret_u8_u32(vdup_n_u32(wd));
|
||||
uint8x8x2_t z01=vzip_u8(vand_u8(by,m2v), vand_u8(vshr_n_u8(by,2),m2v));
|
||||
uint8x8x2_t z23=vzip_u8(vand_u8(vshr_n_u8(by,4),m2v), vshr_n_u8(by,6));
|
||||
uint16x4x2_t zz=vzip_u16(vreinterpret_u16_u8(z01.val[0]), vreinterpret_u16_u8(z23.val[0]));
|
||||
int16x8_t w0=vmovl_s8(vsub_s8(vreinterpret_s8_u16(zz.val[0]),b2v));
|
||||
int16x8_t w1=vmovl_s8(vsub_s8(vreinterpret_s8_u16(zz.val[1]),b2v));
|
||||
ac0=vfmaq_f32(ac0, vld1q_f32(xs+i), vcvtq_f32_s32(vmovl_s16(vget_low_s16(w0))));
|
||||
ac1=vfmaq_f32(ac1, vld1q_f32(xs+i+4), vcvtq_f32_s32(vmovl_s16(vget_high_s16(w0))));
|
||||
ac0=vfmaq_f32(ac0, vld1q_f32(xs+i+8), vcvtq_f32_s32(vmovl_s16(vget_low_s16(w1))));
|
||||
ac1=vfmaq_f32(ac1, vld1q_f32(xs+i+12), vcvtq_f32_s32(vmovl_s16(vget_high_s16(w1)))); }
|
||||
a=vaddvq_f32(vaddq_f32(ac0,ac1));
|
||||
#endif
|
||||
for(;i<I;i++){ uint8_t byte=w[i>>2]; int sh=(i&3)*2; a += xs[i]*(float)((int)((byte>>sh)&3)-2); }
|
||||
y[(int64_t)s*O+o]=a*sc; } }
|
||||
}
|
||||
|
||||
/* ---- IDOT: integer dot kernels (int8-quantized activations) --------------- */
|
||||
#if defined(__AVX512VNNI__) && defined(__AVX512BW__)
|
||||
#define IDOT_KERNEL "avx512-vnni"
|
||||
#elif defined(__AVXVNNI__) && defined(__AVX2__)
|
||||
#define IDOT_KERNEL "avx-vnni"
|
||||
#elif defined(__AVX2__)
|
||||
#define IDOT_KERNEL "avx2"
|
||||
#elif defined(__ARM_NEON) && defined(__ARM_FEATURE_MATMUL_INT8)
|
||||
#define IDOT_KERNEL "neon-i8mm"
|
||||
#elif defined(__ARM_NEON)
|
||||
#define IDOT_KERNEL "neon"
|
||||
#elif defined(__VSX__)
|
||||
#define IDOT_KERNEL "vsx"
|
||||
#else
|
||||
#define IDOT_KERNEL "scalar"
|
||||
#endif
|
||||
static int g_idot=1;
|
||||
#if defined(__ARM_NEON) && defined(__ARM_FEATURE_DOTPROD)
|
||||
static int g_i4s=1;
|
||||
#elif defined(__VSX__)
|
||||
static int g_i4s=1;
|
||||
#else
|
||||
static int g_i4s=2;
|
||||
#endif
|
||||
|
||||
static inline float qrow_i8(const float *x, int8_t *q, int I){
|
||||
float amax=0; for(int i=0;i<I;i++){ float a=fabsf(x[i]); if(a>amax)amax=a; }
|
||||
float s=amax/127.f; if(s<1e-12f) s=1e-12f; float inv=1.f/s;
|
||||
for(int i=0;i<I;i++) q[i]=(int8_t)lrintf(x[i]*inv);
|
||||
return s;
|
||||
}
|
||||
|
||||
/* dot int8*int8 */
|
||||
static inline int32_t dot_i8i8(const int8_t *w, const int8_t *x, int I){
|
||||
int32_t sum=0; int i=0;
|
||||
#if defined(__AVX512VNNI__) && defined(__AVX512BW__)
|
||||
__m512i acc=_mm512_setzero_si512();
|
||||
for(;i+64<=I;i+=64){
|
||||
__m512i wv=_mm512_loadu_si512((const void*)(w+i));
|
||||
__m512i xv=_mm512_loadu_si512((const void*)(x+i));
|
||||
__mmask64 neg=_mm512_movepi8_mask(wv);
|
||||
__m512i xs=_mm512_mask_sub_epi8(xv,neg,_mm512_setzero_si512(),xv);
|
||||
acc=_mm512_dpbusd_epi32(acc,_mm512_abs_epi8(wv),xs);
|
||||
}
|
||||
sum=_mm512_reduce_add_epi32(acc);
|
||||
#elif defined(__AVXVNNI__) && defined(__AVX2__)
|
||||
__m128i acc=_mm_setzero_si128();
|
||||
for(;i+16<=I;i+=16){
|
||||
__m128i wv=_mm_loadu_si128((const __m128i*)(w+i));
|
||||
__m128i xv=_mm_loadu_si128((const __m128i*)(x+i));
|
||||
__m128i xs=_mm_sign_epi8(xv,wv);
|
||||
acc=_mm_dpbusd_epi32(acc,_mm_abs_epi8(wv),xs);
|
||||
}
|
||||
sum=hsum128_i32(acc);
|
||||
#elif defined(__AVX2__)
|
||||
__m256i acc=_mm256_setzero_si256(); const __m256i ones=_mm256_set1_epi16(1);
|
||||
for(;i+32<=I;i+=32){
|
||||
__m256i wv=_mm256_loadu_si256((const __m256i*)(w+i));
|
||||
__m256i xv=_mm256_loadu_si256((const __m256i*)(x+i));
|
||||
__m256i p=_mm256_maddubs_epi16(_mm256_sign_epi8(wv,wv),_mm256_sign_epi8(xv,wv));
|
||||
acc=_mm256_add_epi32(acc,_mm256_madd_epi16(p,ones));
|
||||
}
|
||||
sum=hsum256_i32(acc);
|
||||
#elif defined(__ARM_NEON)
|
||||
#if defined(__ARM_FEATURE_DOTPROD)
|
||||
int32x4_t a0=vdupq_n_s32(0),a1=vdupq_n_s32(0),a2=vdupq_n_s32(0),a3=vdupq_n_s32(0);
|
||||
for(;i+64<=I;i+=64){
|
||||
a0=vdotq_s32(a0,vld1q_s8(w+i), vld1q_s8(x+i));
|
||||
a1=vdotq_s32(a1,vld1q_s8(w+i+16),vld1q_s8(x+i+16));
|
||||
a2=vdotq_s32(a2,vld1q_s8(w+i+32),vld1q_s8(x+i+32));
|
||||
a3=vdotq_s32(a3,vld1q_s8(w+i+48),vld1q_s8(x+i+48));
|
||||
}
|
||||
int32x4_t acc=vaddq_s32(vaddq_s32(a0,a1),vaddq_s32(a2,a3));
|
||||
for(;i+16<=I;i+=16) acc=vdotq_s32(acc,vld1q_s8(w+i),vld1q_s8(x+i));
|
||||
sum=vaddvq_s32(acc);
|
||||
#else
|
||||
int32x4_t acc=vdupq_n_s32(0);
|
||||
for(;i+16<=I;i+=16){
|
||||
int8x16_t wv=vld1q_s8(w+i), xv=vld1q_s8(x+i);
|
||||
int16x8_t p=vmull_s8(vget_low_s8(wv),vget_low_s8(xv));
|
||||
p=vmlal_s8(p,vget_high_s8(wv),vget_high_s8(xv));
|
||||
acc=vpadalq_s16(acc,p);
|
||||
}
|
||||
sum=vaddvq_s32(acc);
|
||||
#endif
|
||||
#elif defined(__VSX__)
|
||||
__vector signed int acc=vec_splats(0);
|
||||
const __vector signed char vz=vec_splats((signed char)0);
|
||||
for(;i+16<=I;i+=16){
|
||||
__vector signed char wv=vec_xl(0,(const signed char*)(w+i));
|
||||
__vector signed char xv=vec_xl(0,(const signed char*)(x+i));
|
||||
__vector __bool char neg=vec_cmplt(wv,vz);
|
||||
__vector signed char xs=vec_sel(xv,vec_sub(vz,xv),neg);
|
||||
__vector unsigned char wa=(__vector unsigned char)vec_sel(wv,vec_sub(vz,wv),neg);
|
||||
acc=vec_msum(xs,wa,acc);
|
||||
}
|
||||
sum=vec_extract(acc,0)+vec_extract(acc,1)+vec_extract(acc,2)+vec_extract(acc,3);
|
||||
#endif
|
||||
for(;i<I;i++) sum+=(int32_t)w[i]*x[i];
|
||||
return sum;
|
||||
}
|
||||
|
||||
/* dot int4(packed)*int8 */
|
||||
static inline int32_t dot_i4i8(const uint8_t *w4, const int8_t *x, int I){
|
||||
int32_t sum=0; int i=0;
|
||||
#if defined(__AVX512VNNI__) && defined(__AVX512BW__)
|
||||
const __m256i m4v=_mm256_set1_epi8(0x0F);
|
||||
const __m512i b8v=_mm512_set1_epi8(8);
|
||||
const __m512i xidx=_mm512_setr_epi64(0,1,4,5,2,3,6,7);
|
||||
__m512i acc=_mm512_setzero_si512();
|
||||
for(;i+64<=I;i+=64){
|
||||
__m256i by=_mm256_loadu_si256((const __m256i*)(w4+(i>>1)));
|
||||
__m256i lo=_mm256_and_si256(by,m4v), hi=_mm256_and_si256(_mm256_srli_epi16(by,4),m4v);
|
||||
__m256i z0=_mm256_unpacklo_epi8(lo,hi), z1=_mm256_unpackhi_epi8(lo,hi);
|
||||
__m512i wv=_mm512_sub_epi8(_mm512_inserti64x4(_mm512_castsi256_si512(z0),z1,1),b8v);
|
||||
__m512i xv=_mm512_permutexvar_epi64(xidx,_mm512_loadu_si512((const void*)(x+i)));
|
||||
__mmask64 neg=_mm512_movepi8_mask(wv);
|
||||
__m512i xs=_mm512_mask_sub_epi8(xv,neg,_mm512_setzero_si512(),xv);
|
||||
acc=_mm512_dpbusd_epi32(acc,_mm512_abs_epi8(wv),xs);
|
||||
}
|
||||
sum=_mm512_reduce_add_epi32(acc);
|
||||
#elif defined(__AVXVNNI__) && defined(__AVX2__)
|
||||
const __m128i m4=_mm_set1_epi8(0x0F); const __m128i b8=_mm_set1_epi8(8);
|
||||
__m128i acc=_mm_setzero_si128();
|
||||
for(;i+32<=I;i+=32){
|
||||
__m128i by=_mm_loadu_si128((const __m128i*)(w4+(i>>1)));
|
||||
__m128i lo=_mm_and_si128(by,m4), hi=_mm_and_si128(_mm_srli_epi16(by,4),m4);
|
||||
__m128i n0=_mm_unpacklo_epi8(lo,hi), n1=_mm_unpackhi_epi8(lo,hi);
|
||||
__m128i w0=_mm_sub_epi8(n0,b8), w1=_mm_sub_epi8(n1,b8);
|
||||
__m128i x0=_mm_loadu_si128((const __m128i*)(x+i));
|
||||
__m128i x1=_mm_loadu_si128((const __m128i*)(x+i+16));
|
||||
acc=_mm_dpbusd_epi32(acc,_mm_abs_epi8(w0),_mm_sign_epi8(x0,w0));
|
||||
acc=_mm_dpbusd_epi32(acc,_mm_abs_epi8(w1),_mm_sign_epi8(x1,w1));
|
||||
}
|
||||
sum=hsum128_i32(acc);
|
||||
#elif defined(__AVX2__)
|
||||
const __m128i m4=_mm_set1_epi8(0x0F); const __m256i b8=_mm256_set1_epi8(8);
|
||||
const __m256i ones=_mm256_set1_epi16(1);
|
||||
__m256i acc=_mm256_setzero_si256();
|
||||
for(;i+32<=I;i+=32){
|
||||
__m128i by=_mm_loadu_si128((const __m128i*)(w4+(i>>1)));
|
||||
__m128i lo=_mm_and_si128(by,m4), hi=_mm_and_si128(_mm_srli_epi16(by,4),m4);
|
||||
__m128i n0=_mm_unpacklo_epi8(lo,hi), n1=_mm_unpackhi_epi8(lo,hi);
|
||||
__m256i wv=_mm256_sub_epi8(_mm256_set_m128i(n1,n0),b8);
|
||||
__m256i xv=_mm256_loadu_si256((const __m256i*)(x+i));
|
||||
__m256i p=_mm256_maddubs_epi16(_mm256_sign_epi8(wv,wv),_mm256_sign_epi8(xv,wv));
|
||||
acc=_mm256_add_epi32(acc,_mm256_madd_epi16(p,ones));
|
||||
}
|
||||
sum=hsum256_i32(acc);
|
||||
#elif defined(__ARM_NEON)
|
||||
const uint8x16_t m4q=vdupq_n_u8(0x0F); const int8x16_t b8q=vdupq_n_s8(8);
|
||||
#if defined(__ARM_FEATURE_DOTPROD)
|
||||
int32x4_t a0=vdupq_n_s32(0),a1=vdupq_n_s32(0),a2=vdupq_n_s32(0),a3=vdupq_n_s32(0);
|
||||
for(;i+64<=I;i+=64){
|
||||
uint8x16_t byA=vld1q_u8(w4+(i>>1)), byB=vld1q_u8(w4+(i>>1)+16);
|
||||
uint8x16x2_t zA=vzipq_u8(vandq_u8(byA,m4q), vshrq_n_u8(byA,4));
|
||||
uint8x16x2_t zB=vzipq_u8(vandq_u8(byB,m4q), vshrq_n_u8(byB,4));
|
||||
a0=vdotq_s32(a0,vsubq_s8(vreinterpretq_s8_u8(zA.val[0]),b8q),vld1q_s8(x+i));
|
||||
a1=vdotq_s32(a1,vsubq_s8(vreinterpretq_s8_u8(zA.val[1]),b8q),vld1q_s8(x+i+16));
|
||||
a2=vdotq_s32(a2,vsubq_s8(vreinterpretq_s8_u8(zB.val[0]),b8q),vld1q_s8(x+i+32));
|
||||
a3=vdotq_s32(a3,vsubq_s8(vreinterpretq_s8_u8(zB.val[1]),b8q),vld1q_s8(x+i+48));
|
||||
}
|
||||
int32x4_t acc=vaddq_s32(vaddq_s32(a0,a1),vaddq_s32(a2,a3));
|
||||
for(;i+32<=I;i+=32){
|
||||
uint8x16_t by=vld1q_u8(w4+(i>>1));
|
||||
uint8x16x2_t z=vzipq_u8(vandq_u8(by,m4q), vshrq_n_u8(by,4));
|
||||
acc=vdotq_s32(acc,vsubq_s8(vreinterpretq_s8_u8(z.val[0]),b8q),vld1q_s8(x+i));
|
||||
acc=vdotq_s32(acc,vsubq_s8(vreinterpretq_s8_u8(z.val[1]),b8q),vld1q_s8(x+i+16));
|
||||
}
|
||||
sum=vaddvq_s32(acc);
|
||||
#else
|
||||
int32x4_t acc=vdupq_n_s32(0);
|
||||
for(;i+32<=I;i+=32){
|
||||
uint8x16_t by=vld1q_u8(w4+(i>>1));
|
||||
uint8x16x2_t z=vzipq_u8(vandq_u8(by,m4q), vshrq_n_u8(by,4));
|
||||
int8x16_t w0=vsubq_s8(vreinterpretq_s8_u8(z.val[0]),b8q);
|
||||
int8x16_t w1=vsubq_s8(vreinterpretq_s8_u8(z.val[1]),b8q);
|
||||
int8x16_t x0=vld1q_s8(x+i), x1=vld1q_s8(x+i+16);
|
||||
int16x8_t p=vmull_s8(vget_low_s8(w0),vget_low_s8(x0));
|
||||
p=vmlal_s8(p,vget_high_s8(w0),vget_high_s8(x0));
|
||||
acc=vpadalq_s16(acc,p);
|
||||
p=vmull_s8(vget_low_s8(w1),vget_low_s8(x1));
|
||||
p=vmlal_s8(p,vget_high_s8(w1),vget_high_s8(x1));
|
||||
acc=vpadalq_s16(acc,p);
|
||||
}
|
||||
sum=vaddvq_s32(acc);
|
||||
#endif
|
||||
#elif defined(__VSX__)
|
||||
const __vector unsigned char m4v=vec_splats((unsigned char)0x0F);
|
||||
const __vector unsigned char sh4=vec_splats((unsigned char)4);
|
||||
const __vector signed char b8v=vec_splats((signed char)8);
|
||||
const __vector signed char vz=vec_splats((signed char)0);
|
||||
__vector signed int acc=vec_splats(0);
|
||||
for(;i+32<=I;i+=32){
|
||||
__vector unsigned char by=vec_xl(0,w4+(i>>1));
|
||||
__vector unsigned char lo=vec_and(by,m4v), hi=vec_sr(by,sh4);
|
||||
__vector signed char w0=vec_sub((__vector signed char)vec_mergeh(lo,hi),b8v);
|
||||
__vector signed char w1=vec_sub((__vector signed char)vec_mergel(lo,hi),b8v);
|
||||
__vector signed char x0=vec_xl(0,(const signed char*)(x+i));
|
||||
__vector signed char x1=vec_xl(0,(const signed char*)(x+i+16));
|
||||
__vector __bool char n0=vec_cmplt(w0,vz), n1=vec_cmplt(w1,vz);
|
||||
acc=vec_msum(vec_sel(x0,vec_sub(vz,x0),n0),
|
||||
(__vector unsigned char)vec_sel(w0,vec_sub(vz,w0),n0),acc);
|
||||
acc=vec_msum(vec_sel(x1,vec_sub(vz,x1),n1),
|
||||
(__vector unsigned char)vec_sel(w1,vec_sub(vz,w1),n1),acc);
|
||||
}
|
||||
sum=vec_extract(acc,0)+vec_extract(acc,1)+vec_extract(acc,2)+vec_extract(acc,3);
|
||||
#endif
|
||||
for(;i+1<I;i+=2){ uint8_t b=w4[i>>1]; sum+=((int)(b&0xF)-8)*x[i]+((int)(b>>4)-8)*x[i+1]; }
|
||||
if(i<I){ uint8_t b=w4[i>>1]; sum+=((int)(b&0xF)-8)*x[i]; }
|
||||
return sum;
|
||||
}
|
||||
|
||||
/* ---- ARM i8mm SMMLA tiled kernels ---------------------------------------- */
|
||||
#if defined(__ARM_NEON) && defined(__ARM_FEATURE_MATMUL_INT8)
|
||||
static inline int32x4_t mm_tile16(int32x4_t acc, int8x16_t wo, int8x16_t wo1,
|
||||
int8x16_t xs, int8x16_t xs1){
|
||||
acc=vmmlaq_s32(acc, vcombine_s8(vget_low_s8(wo), vget_low_s8(wo1)),
|
||||
vcombine_s8(vget_low_s8(xs), vget_low_s8(xs1)));
|
||||
return vmmlaq_s32(acc, vcombine_s8(vget_high_s8(wo), vget_high_s8(wo1)),
|
||||
vcombine_s8(vget_high_s8(xs), vget_high_s8(xs1)));
|
||||
}
|
||||
static void matmul_q_idot_mm(float *y, const int8_t *xq, const float *sx, const int8_t *q,
|
||||
const float *scale, int S, int I, int O){
|
||||
#pragma omp parallel for schedule(static)
|
||||
for(int o=0;o<(O&~1);o+=2){
|
||||
const int8_t *wo=q+(int64_t)o*I, *wo1=q+(int64_t)(o+1)*I;
|
||||
float sc0=scale[o], sc1=scale[o+1];
|
||||
for(int s=0;s<(S&~1);s+=2){
|
||||
const int8_t *xs=xq+(int64_t)s*I, *xs1=xq+(int64_t)(s+1)*I;
|
||||
int32x4_t a0=vdupq_n_s32(0),a1=vdupq_n_s32(0),a2=vdupq_n_s32(0),a3=vdupq_n_s32(0); int i=0;
|
||||
for(;i+64<=I;i+=64){
|
||||
a0=mm_tile16(a0,vld1q_s8(wo+i), vld1q_s8(wo1+i), vld1q_s8(xs+i), vld1q_s8(xs1+i));
|
||||
a1=mm_tile16(a1,vld1q_s8(wo+i+16),vld1q_s8(wo1+i+16),vld1q_s8(xs+i+16),vld1q_s8(xs1+i+16));
|
||||
a2=mm_tile16(a2,vld1q_s8(wo+i+32),vld1q_s8(wo1+i+32),vld1q_s8(xs+i+32),vld1q_s8(xs1+i+32));
|
||||
a3=mm_tile16(a3,vld1q_s8(wo+i+48),vld1q_s8(wo1+i+48),vld1q_s8(xs+i+48),vld1q_s8(xs1+i+48));
|
||||
}
|
||||
for(;i+16<=I;i+=16)
|
||||
a0=mm_tile16(a0,vld1q_s8(wo+i),vld1q_s8(wo1+i),vld1q_s8(xs+i),vld1q_s8(xs1+i));
|
||||
int32x4_t acc=vaddq_s32(vaddq_s32(a0,a1),vaddq_s32(a2,a3));
|
||||
int32_t d00=vgetq_lane_s32(acc,0), d01=vgetq_lane_s32(acc,1);
|
||||
int32_t d10=vgetq_lane_s32(acc,2), d11=vgetq_lane_s32(acc,3);
|
||||
for(;i<I;i++){ int a=wo[i],b=wo1[i],u=xs[i],v=xs1[i];
|
||||
d00+=a*u; d01+=a*v; d10+=b*u; d11+=b*v; }
|
||||
y[(int64_t)s*O+o] =(float)d00*sc0*sx[s];
|
||||
y[(int64_t)s*O+(o+1)] =(float)d10*sc1*sx[s];
|
||||
y[(int64_t)(s+1)*O+o] =(float)d01*sc0*sx[s+1];
|
||||
y[(int64_t)(s+1)*O+(o+1)]=(float)d11*sc1*sx[s+1];
|
||||
}
|
||||
if(S&1){ int s=S-1; const int8_t *xs=xq+(int64_t)s*I;
|
||||
y[(int64_t)s*O+o] =(float)dot_i8i8(wo, xs,I)*sc0*sx[s];
|
||||
y[(int64_t)s*O+(o+1)]=(float)dot_i8i8(wo1,xs,I)*sc1*sx[s]; }
|
||||
}
|
||||
if(O&1){ int o=O-1; const int8_t *w=q+(int64_t)o*I; float sc=scale[o];
|
||||
#pragma omp parallel for schedule(static)
|
||||
for(int s=0;s<S;s++) y[(int64_t)s*O+o]=(float)dot_i8i8(w,xq+(int64_t)s*I,I)*sc*sx[s]; }
|
||||
}
|
||||
static void matmul_i4_idot_mm(float *y, const int8_t *xq, const float *sx, const uint8_t *q4,
|
||||
const float *scale, int S, int I, int O){
|
||||
int rb=(I+1)/2;
|
||||
#pragma omp parallel for schedule(static)
|
||||
for(int o=0;o<(O&~1);o+=2){
|
||||
const uint8x16_t m4q=vdupq_n_u8(0x0F); const int8x16_t b8q=vdupq_n_s8(8);
|
||||
const uint8_t *wo=q4+(int64_t)o*rb, *wo1=q4+(int64_t)(o+1)*rb;
|
||||
float sc0=scale[o], sc1=scale[o+1];
|
||||
for(int s=0;s<(S&~1);s+=2){
|
||||
const int8_t *xs=xq+(int64_t)s*I, *xs1=xq+(int64_t)(s+1)*I;
|
||||
int32x4_t a0=vdupq_n_s32(0),a1=vdupq_n_s32(0),a2=vdupq_n_s32(0),a3=vdupq_n_s32(0); int i=0;
|
||||
for(;i+64<=I;i+=64){
|
||||
uint8x16_t byo=vld1q_u8(wo+(i>>1)), byo1=vld1q_u8(wo1+(i>>1));
|
||||
uint8x16_t cyo=vld1q_u8(wo+(i>>1)+16), cyo1=vld1q_u8(wo1+(i>>1)+16);
|
||||
uint8x16x2_t zo =vzipq_u8(vandq_u8(byo, m4q), vshrq_n_u8(byo, 4));
|
||||
uint8x16x2_t zo1=vzipq_u8(vandq_u8(byo1,m4q), vshrq_n_u8(byo1,4));
|
||||
uint8x16x2_t ko =vzipq_u8(vandq_u8(cyo, m4q), vshrq_n_u8(cyo, 4));
|
||||
uint8x16x2_t ko1=vzipq_u8(vandq_u8(cyo1,m4q), vshrq_n_u8(cyo1,4));
|
||||
a0=mm_tile16(a0, vsubq_s8(vreinterpretq_s8_u8(zo.val[0]),b8q),
|
||||
vsubq_s8(vreinterpretq_s8_u8(zo1.val[0]),b8q),
|
||||
vld1q_s8(xs+i), vld1q_s8(xs1+i));
|
||||
a1=mm_tile16(a1, vsubq_s8(vreinterpretq_s8_u8(zo.val[1]),b8q),
|
||||
vsubq_s8(vreinterpretq_s8_u8(zo1.val[1]),b8q),
|
||||
vld1q_s8(xs+i+16), vld1q_s8(xs1+i+16));
|
||||
a2=mm_tile16(a2, vsubq_s8(vreinterpretq_s8_u8(ko.val[0]),b8q),
|
||||
vsubq_s8(vreinterpretq_s8_u8(ko1.val[0]),b8q),
|
||||
vld1q_s8(xs+i+32), vld1q_s8(xs1+i+32));
|
||||
a3=mm_tile16(a3, vsubq_s8(vreinterpretq_s8_u8(ko.val[1]),b8q),
|
||||
vsubq_s8(vreinterpretq_s8_u8(ko1.val[1]),b8q),
|
||||
vld1q_s8(xs+i+48), vld1q_s8(xs1+i+48));
|
||||
}
|
||||
for(;i+32<=I;i+=32){
|
||||
uint8x16_t byo=vld1q_u8(wo+(i>>1)), byo1=vld1q_u8(wo1+(i>>1));
|
||||
uint8x16x2_t zo =vzipq_u8(vandq_u8(byo, m4q), vshrq_n_u8(byo, 4));
|
||||
uint8x16x2_t zo1=vzipq_u8(vandq_u8(byo1,m4q), vshrq_n_u8(byo1,4));
|
||||
a0=mm_tile16(a0, vsubq_s8(vreinterpretq_s8_u8(zo.val[0]),b8q),
|
||||
vsubq_s8(vreinterpretq_s8_u8(zo1.val[0]),b8q),
|
||||
vld1q_s8(xs+i), vld1q_s8(xs1+i));
|
||||
a1=mm_tile16(a1, vsubq_s8(vreinterpretq_s8_u8(zo.val[1]),b8q),
|
||||
vsubq_s8(vreinterpretq_s8_u8(zo1.val[1]),b8q),
|
||||
vld1q_s8(xs+i+16), vld1q_s8(xs1+i+16));
|
||||
}
|
||||
int32x4_t acc=vaddq_s32(vaddq_s32(a0,a1),vaddq_s32(a2,a3));
|
||||
int32_t d00=vgetq_lane_s32(acc,0), d01=vgetq_lane_s32(acc,1);
|
||||
int32_t d10=vgetq_lane_s32(acc,2), d11=vgetq_lane_s32(acc,3);
|
||||
for(;i+1<I;i+=2){ uint8_t bo=wo[i>>1], bo1=wo1[i>>1];
|
||||
int a0=(int)(bo&0xF)-8, a1=(int)(bo>>4)-8, b0=(int)(bo1&0xF)-8, b1=(int)(bo1>>4)-8;
|
||||
int u0=xs[i],u1=xs[i+1],v0=xs1[i],v1=xs1[i+1];
|
||||
d00+=a0*u0+a1*u1; d01+=a0*v0+a1*v1; d10+=b0*u0+b1*u1; d11+=b0*v0+b1*v1; }
|
||||
if(i<I){ uint8_t bo=wo[i>>1], bo1=wo1[i>>1];
|
||||
int a0=(int)(bo&0xF)-8, b0=(int)(bo1&0xF)-8;
|
||||
d00+=a0*xs[i]; d01+=a0*xs1[i]; d10+=b0*xs[i]; d11+=b0*xs1[i]; }
|
||||
y[(int64_t)s*O+o] =(float)d00*sc0*sx[s];
|
||||
y[(int64_t)s*O+(o+1)] =(float)d10*sc1*sx[s];
|
||||
y[(int64_t)(s+1)*O+o] =(float)d01*sc0*sx[s+1];
|
||||
y[(int64_t)(s+1)*O+(o+1)]=(float)d11*sc1*sx[s+1];
|
||||
}
|
||||
if(S&1){ int s=S-1; const int8_t *xs=xq+(int64_t)s*I;
|
||||
y[(int64_t)s*O+o] =(float)dot_i4i8(wo, xs,I)*sc0*sx[s];
|
||||
y[(int64_t)s*O+(o+1)]=(float)dot_i4i8(wo1,xs,I)*sc1*sx[s]; }
|
||||
}
|
||||
if(O&1){ int o=O-1; const uint8_t *w=q4+(int64_t)o*rb; float sc=scale[o];
|
||||
#pragma omp parallel for schedule(static)
|
||||
for(int s=0;s<S;s++) y[(int64_t)s*O+o]=(float)dot_i4i8(w,xq+(int64_t)s*I,I)*sc*sx[s]; }
|
||||
}
|
||||
#endif
|
||||
|
||||
/* ---- IDOT dispatch (int8-quantized activations) --------------------------- */
|
||||
static void matmul_q_idot(float *y, const int8_t *xq, const float *sx, const int8_t *q,
|
||||
const float *scale, int S, int I, int O){
|
||||
#if defined(__ARM_NEON) && defined(__ARM_FEATURE_MATMUL_INT8)
|
||||
if(S>=2){ matmul_q_idot_mm(y,xq,sx,q,scale,S,I,O); return; }
|
||||
#endif
|
||||
#pragma omp parallel for schedule(static)
|
||||
for(int o=0;o<O;o++){ const int8_t *w=q+(int64_t)o*I; float sc=scale[o];
|
||||
for(int s=0;s<S;s++) y[(int64_t)s*O+o]=(float)dot_i8i8(w,xq+(int64_t)s*I,I)*sc*sx[s]; }
|
||||
}
|
||||
static void matmul_i4_idot(float *y, const int8_t *xq, const float *sx, const uint8_t *q4,
|
||||
const float *scale, int S, int I, int O){
|
||||
int rb=(I+1)/2;
|
||||
#if defined(__ARM_NEON) && defined(__ARM_FEATURE_MATMUL_INT8)
|
||||
if(S>=2){ matmul_i4_idot_mm(y,xq,sx,q4,scale,S,I,O); return; }
|
||||
#endif
|
||||
#pragma omp parallel for schedule(static)
|
||||
for(int o=0;o<O;o++){ const uint8_t *w=q4+(int64_t)o*rb; float sc=scale[o];
|
||||
for(int s=0;s<S;s++) y[(int64_t)s*O+o]=(float)dot_i4i8(w,xq+(int64_t)s*I,I)*sc*sx[s]; }
|
||||
}
|
||||
|
||||
/* ---- per-thread quantization scratch -------------------------------------- */
|
||||
typedef struct { int8_t *xq; size_t xq_cap; float *sx; size_t sx_cap; } QScratch;
|
||||
static _Thread_local QScratch g_qscratch;
|
||||
static void quant_scratch(size_t xn, size_t sn, int8_t **xq, float **sx){
|
||||
if(xn>g_qscratch.xq_cap){
|
||||
int8_t *p=realloc(g_qscratch.xq,xn);
|
||||
if(!p){ fprintf(stderr,"OOM quant scratch\n"); exit(1); }
|
||||
g_qscratch.xq=p; g_qscratch.xq_cap=xn;
|
||||
}
|
||||
if(sn>g_qscratch.sx_cap){
|
||||
float *p=realloc(g_qscratch.sx,sn*sizeof(float));
|
||||
if(!p){ fprintf(stderr,"OOM quant scales\n"); exit(1); }
|
||||
g_qscratch.sx=p; g_qscratch.sx_cap=sn;
|
||||
}
|
||||
*xq=g_qscratch.xq; *sx=g_qscratch.sx;
|
||||
}
|
||||
|
||||
/* ---- f32 -> quantized packing --------------------------------------------- */
|
||||
static void quantize_rows(const float *w, int8_t *q, float *scale, int O, int I, int bits){
|
||||
int qmax=(1<<(bits-1))-1;
|
||||
#pragma omp parallel for schedule(static)
|
||||
for(int o=0;o<O;o++){ const float *wr=w+(int64_t)o*I; float amax=0;
|
||||
for(int i=0;i<I;i++){ float a=fabsf(wr[i]); if(a>amax)amax=a; }
|
||||
float s=amax/qmax; if(s<1e-8f)s=1e-8f; scale[o]=s;
|
||||
int8_t *qr=q+(int64_t)o*I;
|
||||
for(int i=0;i<I;i++){ int v=(int)lrintf(wr[i]/s); if(v>qmax)v=qmax; if(v<-qmax-1)v=-qmax-1; qr[i]=(int8_t)v; }
|
||||
}
|
||||
}
|
||||
static void pack_int4(const float *w, uint8_t *q4, float *scale, int O, int I, int bits){
|
||||
int qmax=(1<<(bits-1))-1, rb=(I+1)/2;
|
||||
#pragma omp parallel for schedule(static)
|
||||
for(int o=0;o<O;o++){ const float *wr=w+(int64_t)o*I; float amax=0;
|
||||
for(int i=0;i<I;i++){ float a=fabsf(wr[i]); if(a>amax)amax=a; }
|
||||
float s=amax/qmax; if(s<1e-8f)s=1e-8f; scale[o]=s;
|
||||
uint8_t *qr=q4+(int64_t)o*rb;
|
||||
for(int i=0;i<I;i+=2){
|
||||
int v0=(int)lrintf(wr[i]/s); if(v0>qmax)v0=qmax; if(v0<-8)v0=-8;
|
||||
int v1=0; if(i+1<I){ v1=(int)lrintf(wr[i+1]/s); if(v1>qmax)v1=qmax; if(v1<-8)v1=-8; }
|
||||
qr[i>>1] = (uint8_t)((v0+8) | ((v1+8)<<4));
|
||||
}
|
||||
}
|
||||
}
|
||||
static void pack_int2(const float *w, uint8_t *q2, float *scale, int O, int I, int bits){
|
||||
int qmax=(1<<(bits-1))-1, rb=(I+3)/4;
|
||||
#pragma omp parallel for schedule(static)
|
||||
for(int o=0;o<O;o++){ const float *wr=w+(int64_t)o*I; float amax=0;
|
||||
for(int i=0;i<I;i++){ float a=fabsf(wr[i]); if(a>amax)amax=a; }
|
||||
float s=amax/qmax; if(s<1e-8f)s=1e-8f; scale[o]=s;
|
||||
uint8_t *qr=q2+(int64_t)o*rb;
|
||||
for(int i=0;i<I;i+=4){ uint8_t byte=0;
|
||||
for(int k=0;k<4 && i+k<I;k++){ int v=(int)lrintf(wr[i+k]/s); if(v>qmax)v=qmax; if(v<-2)v=-2; byte|=(uint8_t)((v+2)<<(k*2)); }
|
||||
qr[i>>2]=byte;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#endif /* COLI_QUANT_H */
|
||||
+143
@@ -0,0 +1,143 @@
|
||||
/* sample.h — sampling (temperature + nucleus) and stop-set management.
|
||||
* Header-only: all functions are static — include from the main engine file. */
|
||||
#ifndef SAMPLE_H
|
||||
#define SAMPLE_H
|
||||
|
||||
#include <math.h>
|
||||
#include <stdio.h>
|
||||
#include <string.h>
|
||||
#include "tok.h"
|
||||
|
||||
/* ---- RNG (xorshift64*) -------------------------------------------------- */
|
||||
static uint64_t g_rng = 0x9E3779B97F4A7C15ULL;
|
||||
static inline double rndu(void){
|
||||
g_rng ^= g_rng << 13; g_rng ^= g_rng >> 7; g_rng ^= g_rng << 17;
|
||||
return (double)(g_rng >> 11) * (1.0 / 9007199254740992.0);
|
||||
}
|
||||
|
||||
/* ---- argmax over a float vector ----------------------------------------- */
|
||||
static inline int argmax_v(const float *lo, int V){
|
||||
int b=-1; float bv=-INFINITY;
|
||||
for(int i=0;i<V;i++){ float x=lo[i]; if(x==x && x>bv){ bv=x; b=i; } }
|
||||
return b<0?0:b;
|
||||
}
|
||||
|
||||
/* ---- distribution buffers (reused, single-threaded decode) --------------- */
|
||||
static float *g_pbuf = NULL;
|
||||
static int *g_pidx = NULL;
|
||||
|
||||
/* sift-down on max-heap in h[0..n), key = g_pbuf[h[i]] (#335: partial top-p).
|
||||
* "hole" variant: carries the root value and deposits only at the end, so
|
||||
* heapify is O(V) and each pop is O(log n) without qsort on the full vocab. */
|
||||
static void topp_siftdown(int *h, int n, int i){
|
||||
int iv = h[i]; float kv = g_pbuf[iv];
|
||||
for (;;) {
|
||||
int l = 2*i + 1;
|
||||
if (l >= n) break;
|
||||
int b = l; if (l+1 < n && g_pbuf[h[l+1]] > g_pbuf[h[l]]) b = l+1;
|
||||
if (g_pbuf[h[b]] <= kv) break;
|
||||
h[i] = h[b]; i = b;
|
||||
}
|
||||
h[i] = iv;
|
||||
}
|
||||
|
||||
/* build the target distribution in g_pbuf: softmax(lo/temp) truncated to
|
||||
* top-p g_nuc. Invariant: g_pbuf stays indexed by token-id (never reordered);
|
||||
* the truncated tail is zeroed (dist_sample reads by id directly).
|
||||
* Requires: g_temp, g_nuc, falloc() — declared in the main engine file. */
|
||||
static void dist_build(const float *lo, int V){
|
||||
if (!g_pbuf) { g_pbuf = falloc(V); g_pidx = malloc(V * sizeof(int)); }
|
||||
int mxi = -1; float mx = 0;
|
||||
for (int i = 0; i < V; i++)
|
||||
if (isfinite(lo[i]) && (mxi < 0 || lo[i] > mx)) { mx = lo[i]; mxi = i; }
|
||||
double s = 0; float invt = 1.f / (g_temp > 1e-4f ? g_temp : 1e-4f);
|
||||
if (mxi >= 0) {
|
||||
for (int i = 0; i < V; i++) {
|
||||
g_pbuf[i] = isfinite(lo[i]) ? expf((lo[i] - mx) * invt) : 0.f;
|
||||
s += g_pbuf[i];
|
||||
}
|
||||
}
|
||||
if (mxi < 0 || !isfinite(s) || s <= 0.0) {
|
||||
static int warned = 0;
|
||||
if (!warned) { warned = 1; fprintf(stderr,
|
||||
"[SAMPLE] warning: non-finite logits (NaN/Inf) — falling back to argmax; "
|
||||
"output may be degraded. This usually means a numerical blow-up upstream.\n"); }
|
||||
int a = (mxi >= 0) ? mxi : 0;
|
||||
for (int i = 0; i < V; i++) g_pbuf[i] = 0.f;
|
||||
g_pbuf[a] = 1.f;
|
||||
return;
|
||||
}
|
||||
for (int i = 0; i < V; i++) g_pbuf[i] /= (float)s;
|
||||
if (g_nuc > 0 && g_nuc < 1.f) {
|
||||
for (int i = 0; i < V; i++) g_pidx[i] = i;
|
||||
for (int i = V/2-1; i >= 0; i--) topp_siftdown(g_pidx, V, i);
|
||||
double s2 = 0, cum = 0; int out = V;
|
||||
do {
|
||||
int root = g_pidx[0];
|
||||
g_pidx[0] = g_pidx[--out]; g_pidx[out] = root;
|
||||
s2 += g_pbuf[root]; cum += g_pbuf[root];
|
||||
if (out > 0) topp_siftdown(g_pidx, out, 0);
|
||||
} while (cum < g_nuc && out > 0);
|
||||
for (int i = 0; i < out; i++) g_pbuf[g_pidx[i]] = 0;
|
||||
float s2f = (float)s2;
|
||||
for (int i = out; i < V; i++) g_pbuf[g_pidx[i]] /= s2f;
|
||||
}
|
||||
}
|
||||
|
||||
/* sample from g_pbuf; ban>=0 excludes that token (renormalizing on the fly) */
|
||||
static int dist_sample(int V, int ban){
|
||||
double z = 1.0 - (ban >= 0 ? g_pbuf[ban] : 0.0);
|
||||
if (z <= 1e-12) z = 1e-12;
|
||||
double u = rndu() * z, cum = 0;
|
||||
for (int i = 0; i < V; i++) { if (i == ban) continue; cum += g_pbuf[i]; if (cum >= u) return i; }
|
||||
for (int i = V-1; i >= 0; i--) if (i != ban && g_pbuf[i] > 0) return i;
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* next token from logits: greedy if g_temp<=0, sampling otherwise.
|
||||
* ban = token excluded because it was rejected by speculative verification. */
|
||||
static int pick_tok(const float *lo, int V, int ban){
|
||||
if (g_temp <= 0) return argmax_v(lo, V);
|
||||
dist_build(lo, V);
|
||||
return dist_sample(V, ban);
|
||||
}
|
||||
|
||||
/* ---- stop set ----------------------------------------------------------- */
|
||||
static int g_stop[64], g_nstop = 0;
|
||||
static inline int is_stop(int t){
|
||||
for (int i = 0; i < g_nstop; i++) if (t == g_stop[i]) return 1;
|
||||
return 0;
|
||||
}
|
||||
/* T=NULL -> config stops only (validation/oracle, where the tokenizer is not needed). */
|
||||
static void stops_arm_tok(const Cfg *c, int tok_eos, Tok *T){
|
||||
g_nstop = 0;
|
||||
for (int i = 0; i < c->n_stop && g_nstop < 64; i++) g_stop[g_nstop++] = c->stop_ids[i];
|
||||
if (tok_eos >= 0 && !is_stop(tok_eos) && g_nstop < 64) g_stop[g_nstop++] = tok_eos;
|
||||
int nsp = 0;
|
||||
if (T) for (int id = 0; id < T->n_ids && g_nstop < 64; id++)
|
||||
if (T->id_special[id] && !is_stop(id)) { g_stop[g_nstop++] = id; nsp++; }
|
||||
fprintf(stderr, "[stop] %d stop tokens:", g_nstop);
|
||||
for (int i = 0; i < g_nstop; i++) fprintf(stderr, " %d", g_stop[i]);
|
||||
if (nsp) fprintf(stderr, " (%d from the tokenizer's special set)", nsp);
|
||||
fprintf(stderr, "\n");
|
||||
}
|
||||
static void stops_arm(const Cfg *c, int tok_eos){ stops_arm_tok(c, tok_eos, NULL); }
|
||||
|
||||
/* ---- log-prob of a target token given the logit vector ------------------- */
|
||||
static double logprob_target(const float *lo, int V, int target, int *am){
|
||||
float mx = lo[0]; int best = 0;
|
||||
for (int i = 1; i < V; i++) if (lo[i] > mx) { mx = lo[i]; best = i; }
|
||||
double se = 0;
|
||||
for (int i = 0; i < V; i++) se += exp((double)lo[i] - mx);
|
||||
if (am) *am = (best == target);
|
||||
return (double)(lo[target] - mx) - log(se);
|
||||
}
|
||||
|
||||
/* "glm" in model_type, case-insensitive */
|
||||
static int mt_is_glm(const char *s){
|
||||
if (s) for (; *s; s++)
|
||||
if ((s[0]|32) == 'g' && (s[1]|32) == 'l' && (s[2]|32) == 'm') return 1;
|
||||
return 0;
|
||||
}
|
||||
|
||||
#endif /* SAMPLE_H */
|
||||
+2
-2
@@ -32,11 +32,11 @@ esac
|
||||
|
||||
# 2) build: nativa (veloce, per QUESTA macchina). Per un binario da distribuire: make portable
|
||||
echo " building (ARCH=${ARCH:-native})…"
|
||||
make -s glm ARCH="${ARCH:-native}"
|
||||
make -s colibri ARCH="${ARCH:-native}"
|
||||
|
||||
# 3) self-test sull'oracolo tiny, se presente
|
||||
if [ -d glm_tiny ] && [ -f ref_glm.json ]; then
|
||||
r=$(SNAP=./glm_tiny TF=1 ./glm 64 16 16 2>/dev/null | grep -oE "[0-9]+/[0-9]+ positions" || true)
|
||||
r=$(SNAP=./glm_tiny TF=1 ./colibri 64 16 16 2>/dev/null | grep -oE "[0-9]+/[0-9]+ positions" || true)
|
||||
echo " engine self-test: ${r:-?} (expected 32/32)"
|
||||
fi
|
||||
|
||||
|
||||
+189
@@ -0,0 +1,189 @@
|
||||
/* telemetry.h — dashboard protocol lines, stats/usage persistence, hardware probe.
|
||||
* Include after Model/Cfg/QT/ESlot/shards and st.h are defined; requires
|
||||
* qt_bytes(), now_s(), rss_gb(), edisk_s(), and the g_cuda_* globals (ifdef). */
|
||||
#ifndef TELEMETRY_H
|
||||
#define TELEMETRY_H
|
||||
|
||||
static int64_t tbytes(int O,int I,int bits){
|
||||
if(bits>=16) return (int64_t)O*I*4;
|
||||
if(bits>=5) return (int64_t)O*I + (int64_t)O*4;
|
||||
return (int64_t)O*((I+1)/2) + (int64_t)O*4;
|
||||
}
|
||||
|
||||
static int64_t expert_bytes_probe(Model *m, int ebits){
|
||||
Cfg *c=&m->c; int64_t eb=0; char nm[256];
|
||||
snprintf(nm,sizeof(nm),"model.layers.%d.mlp.experts.0.gate_proj.weight",c->first_dense);
|
||||
if(st_nbytes(&m->S,nm)>0){
|
||||
const char *suf[3]={"gate_proj","up_proj","down_proj"};
|
||||
for(int k=0;k<3;k++){
|
||||
snprintf(nm,sizeof(nm),"model.layers.%d.mlp.experts.0.%s.weight",c->first_dense,suf[k]);
|
||||
eb+=st_nbytes(&m->S,nm);
|
||||
snprintf(nm,sizeof(nm),"model.layers.%d.mlp.experts.0.%s.weight.qs",c->first_dense,suf[k]);
|
||||
int64_t q=st_nbytes(&m->S,nm); if(q>0) eb+=q;
|
||||
}
|
||||
}
|
||||
if(eb<=0) eb = tbytes(c->moe_inter,c->hidden,ebits)*2 + tbytes(c->hidden,c->moe_inter,ebits);
|
||||
return eb;
|
||||
}
|
||||
|
||||
/* BRAIN MAP: per-turn expert hit bitmap for the dashboard. */
|
||||
static uint8_t **g_ehit;
|
||||
static void ehit_mark(Model *m, int layer, int eid){
|
||||
if(!g_ehit){ Cfg *c=&m->c;
|
||||
g_ehit=calloc(c->n_layers+1,sizeof(uint8_t*));
|
||||
for(int i=0;i<=c->n_layers;i++) g_ehit[i]=calloc(c->n_experts,1);
|
||||
}
|
||||
g_ehit[layer][eid]=1;
|
||||
}
|
||||
|
||||
/* CPU model + cores + RAM (GB); empty/zero where unavailable. */
|
||||
static void hw_probe(char *cpu, size_t cn, int *cores, double *ram_total, double *ram_avail){
|
||||
cpu[0]=0;
|
||||
#ifdef _WIN32
|
||||
#if defined(__x86_64__) || defined(__i386__)
|
||||
{ unsigned int r[12]={0}; unsigned int *w=r;
|
||||
for(unsigned int f=0x80000002u; f<=0x80000004u; f++,w+=4)
|
||||
__get_cpuid(f,&w[0],&w[1],&w[2],&w[3]);
|
||||
char *b=(char*)r; b[47]=0; while(*b==' ')b++;
|
||||
snprintf(cpu,cn,"%s",b); }
|
||||
#endif
|
||||
#else
|
||||
FILE *ci=fopen("/proc/cpuinfo","r");
|
||||
if(ci){ char ln[256];
|
||||
while(fgets(ln,sizeof(ln),ci)) if(!strncmp(ln,"model name",10)){
|
||||
char *p=strchr(ln,':'); if(p){ p++; while(*p==' ')p++;
|
||||
int n=(int)strlen(p); if(n>0&&p[n-1]=='\n')p[--n]=0;
|
||||
snprintf(cpu,cn,"%s",p); } break; }
|
||||
fclose(ci); }
|
||||
#endif
|
||||
*cores=0;
|
||||
#ifdef _WIN32
|
||||
{ SYSTEM_INFO si; GetSystemInfo(&si); *cores=(int)si.dwNumberOfProcessors; }
|
||||
#elif defined(_SC_NPROCESSORS_ONLN)
|
||||
*cores=(int)sysconf(_SC_NPROCESSORS_ONLN);
|
||||
#endif
|
||||
*ram_total=*ram_avail=0;
|
||||
#ifdef _WIN32
|
||||
compat_meminfo(ram_total,ram_avail);
|
||||
#else
|
||||
FILE *mi=fopen("/proc/meminfo","r");
|
||||
if(mi){ char ln[256]; double mt=0,ma=0;
|
||||
while(fgets(ln,sizeof(ln),mi)){
|
||||
if(sscanf(ln,"MemTotal: %lf",&mt)==1) *ram_total=mt/1e6;
|
||||
if(sscanf(ln,"MemAvailable: %lf",&ma)==1) *ram_avail=ma/1e6;
|
||||
} fclose(mi); }
|
||||
#endif
|
||||
}
|
||||
|
||||
static void hwinfo_emit(Model *m){
|
||||
Cfg *c=&m->c; (void)c;
|
||||
char cpu[256]; int cores; double ram_total,ram_avail;
|
||||
hw_probe(cpu,sizeof(cpu),&cores,&ram_total,&ram_avail);
|
||||
int ngpu=0; double vram_total=0;
|
||||
char gpu_name[128]="";
|
||||
#ifdef COLI_CUDA
|
||||
ngpu=g_cuda_ndev; vram_total=m->gpu_expert_bytes/1e9;
|
||||
for(int i=0;i<g_cuda_ndev;i++){
|
||||
size_t fr=0,to=0; coli_cuda_mem_info(g_cuda_devices[i],&fr,&to);
|
||||
if(!i) vram_total=(double)to*g_cuda_ndev/1e9;
|
||||
}
|
||||
if(g_cuda_ndev>0)
|
||||
snprintf(gpu_name,sizeof(gpu_name),"CUDA device x%d",g_cuda_ndev);
|
||||
#endif
|
||||
printf("HWINFO %d %.1f %.1f %d %.1f %s|%s\n",
|
||||
cores,ram_total,ram_avail,ngpu,vram_total,cpu,gpu_name);
|
||||
fflush(stdout);
|
||||
}
|
||||
|
||||
static void tiers_emit(Model *m){
|
||||
Cfg *c=&m->c; int nsp=0;
|
||||
for(int i=0;i<c->n_layers;i++) if(m->L[i].sparse) nsp++;
|
||||
int total=(nsp+(m->has_mtp?1:0))*c->n_experts;
|
||||
int pinned=0,lru=0;
|
||||
for(int i=0;i<=c->n_layers;i++){ pinned+=m->npin?m->npin[i]:0; lru+=m->ecn?m->ecn[i]:0; }
|
||||
int vram=0; double vram_gb=0;
|
||||
#ifdef COLI_CUDA
|
||||
vram=m->gpu_expert_count; vram_gb=m->gpu_expert_bytes/1e9;
|
||||
#endif
|
||||
int ram=pinned-vram+lru; if(ram<0) ram=0;
|
||||
int disk=total-vram-ram; if(disk<0) disk=0;
|
||||
double eb=(double)expert_bytes_probe(m,m->ebits);
|
||||
printf("TIERS %d %d %d %.2f %.2f\n",vram,ram,disk,vram_gb,ram*eb/1e9);
|
||||
fflush(stdout);
|
||||
}
|
||||
|
||||
static void emap_emit(Model *m){
|
||||
Cfg *c=&m->c;
|
||||
int rows=0;
|
||||
for(int i=0;i<c->n_layers;i++) if(m->L[i].sparse) rows++;
|
||||
int has_mtp = m->has_mtp && m->eusage[c->n_layers];
|
||||
if(has_mtp) rows++;
|
||||
int cols=c->n_experts;
|
||||
char *hex=malloc((size_t)rows*cols*2+1); int w=0;
|
||||
for(int i=0;i<=c->n_layers;i++){
|
||||
int is_row = (i<c->n_layers && m->L[i].sparse) || (i==c->n_layers && has_mtp);
|
||||
if(!is_row) continue;
|
||||
for(int e=0;e<cols;e++){
|
||||
int tier=0;
|
||||
ESlot *P=m->pin[i];
|
||||
for(int z=0;z<m->npin[i];z++) if(P[z].eid==e){
|
||||
#ifdef COLI_CUDA
|
||||
tier = P[z].g.cuda?2:1;
|
||||
#else
|
||||
tier = 1;
|
||||
#endif
|
||||
break; }
|
||||
if(!tier && m->ecache && m->ecache[i])
|
||||
for(int z=0;z<m->ecn[i];z++) if(m->ecache[i][z].eid==e){ tier=1; break; }
|
||||
uint32_t u = m->eusage[i]?m->eusage[i][e]:0;
|
||||
int heat=0; while(u){ heat++; u>>=1; } if(heat>63) heat=63;
|
||||
int b=(tier<<6)|heat;
|
||||
hex[w++]="0123456789abcdef"[b>>4]; hex[w++]="0123456789abcdef"[b&15];
|
||||
}
|
||||
}
|
||||
hex[w]=0;
|
||||
printf("EMAP %d %d %s\n",rows,cols,hex); fflush(stdout); free(hex);
|
||||
}
|
||||
|
||||
static void hits_emit(Model *m){
|
||||
Cfg *c=&m->c; if(!g_ehit) return;
|
||||
int rows=0;
|
||||
for(int i=0;i<c->n_layers;i++) if(m->L[i].sparse) rows++;
|
||||
int has_mtp = m->has_mtp && m->eusage[c->n_layers];
|
||||
if(has_mtp) rows++;
|
||||
int cols=c->n_experts, nb=(rows*cols+7)/8;
|
||||
uint8_t *bm=calloc(nb,1); int bit=0;
|
||||
for(int i=0;i<=c->n_layers;i++){
|
||||
int is_row = (i<c->n_layers && m->L[i].sparse) || (i==c->n_layers && has_mtp);
|
||||
if(!is_row) continue;
|
||||
for(int e=0;e<cols;e++,bit++)
|
||||
if(g_ehit[i][e]){ bm[bit>>3]|=1<<(bit&7); g_ehit[i][e]=0; }
|
||||
}
|
||||
char *hex=malloc((size_t)nb*2+1); int w=0;
|
||||
for(int b=0;b<nb;b++){ hex[w++]="0123456789abcdef"[bm[b]>>4]; hex[w++]="0123456789abcdef"[bm[b]&15]; }
|
||||
hex[w]=0;
|
||||
printf("HITS %d %d %s\n",rows,cols,hex); fflush(stdout); free(hex); free(bm);
|
||||
}
|
||||
|
||||
static void stats_dump_q(Model *m, const char *path, int quiet){
|
||||
char tmp[2100]; snprintf(tmp,sizeof(tmp),"%s.tmp",path);
|
||||
FILE *f=fopen(tmp,"w"); if(!f){ if(!quiet) perror(tmp); return; }
|
||||
Cfg *c=&m->c; int64_t tot=0, nz=0;
|
||||
for(int i=0;i<=c->n_layers;i++){ if(!m->eusage[i]) continue;
|
||||
for(int e=0;e<c->n_experts;e++) if(m->eusage[i][e]){ fprintf(f,"%d %d %u\n",i,e,m->eusage[i][e]); tot+=m->eusage[i][e]; nz++; } }
|
||||
fclose(f); rename(tmp,path);
|
||||
if(!quiet) fprintf(stderr,"[STATS] %lld selections across %lld distinct experts -> %s\n",(long long)tot,(long long)nz,path);
|
||||
}
|
||||
static void stats_dump(Model *m, const char *path){ stats_dump_q(m,path,0); }
|
||||
|
||||
static char g_usage_path[2100]="";
|
||||
static int64_t usage_load(Model *m, const char *path){
|
||||
FILE *f=fopen(path,"r"); if(!f) return 0;
|
||||
Cfg *c=&m->c; int l,e; uint32_t cnt; int64_t tot=0;
|
||||
while(fscanf(f,"%d %d %u",&l,&e,&cnt)==3)
|
||||
if(l>=0&&l<=c->n_layers&&e>=0&&e<c->n_experts&&m->eusage[l]){ m->eusage[l][e]+=cnt; tot+=cnt; }
|
||||
fclose(f); return tot;
|
||||
}
|
||||
static void usage_save(Model *m){ if(g_usage_path[0]) stats_dump_q(m,g_usage_path,1); }
|
||||
|
||||
#endif /* TELEMETRY_H */
|
||||
@@ -24,7 +24,7 @@
|
||||
* Run: make tests/bench_dsa_select && ./tests/bench_dsa_select (not in TEST_BINS)
|
||||
*/
|
||||
#define main coli_glm_main_unused
|
||||
#include "../glm.c"
|
||||
#include "../colibri.c"
|
||||
#undef main
|
||||
|
||||
#include <math.h>
|
||||
|
||||
@@ -22,7 +22,7 @@
|
||||
* Run: make tests/bench_topp && ./tests/bench_topp (not in TEST_BINS -- not a gate)
|
||||
*/
|
||||
#define main coli_glm_main_unused
|
||||
#include "../glm.c"
|
||||
#include "../colibri.c"
|
||||
#undef main
|
||||
|
||||
#include <math.h>
|
||||
|
||||
@@ -31,7 +31,7 @@
|
||||
* In-memory only (no scratch files), so it builds clean on the Windows MinGW CI
|
||||
* job without the unmerged compat shim. */
|
||||
#define main coli_glm_main_unused
|
||||
#include "../glm.c"
|
||||
#include "../colibri.c"
|
||||
#undef main
|
||||
|
||||
#include <math.h>
|
||||
|
||||
@@ -18,7 +18,7 @@
|
||||
* index, a wrong group boundary or a swapped nibble cannot hide under it —
|
||||
* those are O(1) relative errors, not O(1e-6). */
|
||||
#define main coli_glm_main_unused
|
||||
#include "../glm.c"
|
||||
#include "../colibri.c"
|
||||
#undef main
|
||||
|
||||
static uint32_t rng_state=0xC0FFEEu;
|
||||
|
||||
+1
-1
@@ -7,7 +7,7 @@
|
||||
* (sign-trick kernels must treat |−128| as 128 unsigned, not saturate to 127),
|
||||
* and random data at qrow_i8's contract (|x| <= 127, w full int8 range). */
|
||||
#define main coli_glm_main_unused
|
||||
#include "../glm.c"
|
||||
#include "../colibri.c"
|
||||
#undef main
|
||||
|
||||
static uint32_t rng_state=0x12345678u;
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
* arrays twice on the second call -> allocator abort. No model file needed:
|
||||
* the CPU path of kv_alloc only reads c->n_layers/kv_lora/qk_rope. */
|
||||
#define main coli_glm_main_unused
|
||||
#include "../glm.c"
|
||||
#include "../colibri.c"
|
||||
#undef main
|
||||
|
||||
int main(void){
|
||||
|
||||
@@ -15,7 +15,7 @@
|
||||
#include <assert.h>
|
||||
#include <math.h>
|
||||
#define main coli_glm_main_unused
|
||||
#include "../glm.c"
|
||||
#include "../colibri.c"
|
||||
#undef main
|
||||
|
||||
static int approx1(double x){ return x > 0.999 && x < 1.001; }
|
||||
|
||||
@@ -35,7 +35,7 @@ class MakefilePlatformTests(unittest.TestCase):
|
||||
env["PATH"] = ""
|
||||
|
||||
result = subprocess.run(
|
||||
[MAKE, "--no-print-directory", "-B", "-n", "glm"],
|
||||
[MAKE, "--no-print-directory", "-B", "-n", "colibri"],
|
||||
cwd=C_DIR,
|
||||
env=env,
|
||||
text=True,
|
||||
@@ -43,7 +43,7 @@ class MakefilePlatformTests(unittest.TestCase):
|
||||
check=True,
|
||||
)
|
||||
|
||||
self.assertIn("-o glm.exe", result.stdout)
|
||||
self.assertIn("-o colibri.exe", result.stdout)
|
||||
self.assertIn("-fopenmp", result.stdout)
|
||||
self.assertIn("-static", result.stdout)
|
||||
|
||||
|
||||
@@ -6,7 +6,7 @@
|
||||
* iniettato il token scelto e' l'argmax dei FINITI (mai 0 per default), su ogni posizione
|
||||
* del NaN inclusa lo[0]; (c) nessun NaN sopravvive in g_pbuf. */
|
||||
#define main coli_glm_main_unused
|
||||
#include "../glm.c"
|
||||
#include "../colibri.c"
|
||||
#undef main
|
||||
#include <stdio.h>
|
||||
|
||||
|
||||
@@ -20,7 +20,7 @@
|
||||
* Defense 2 is what makes this robust against checkpoints we don't control:
|
||||
* even with BOTH configs mutilated, a control token cannot leak into a reply. */
|
||||
#define main coli_glm_main_unused
|
||||
#include "../glm.c"
|
||||
#include "../colibri.c"
|
||||
#undef main
|
||||
|
||||
static const char *TOKJSON =
|
||||
|
||||
+1
-1
@@ -24,7 +24,7 @@
|
||||
* No scratch files: the test runs entirely in memory (no mkdtemp), so it builds clean on
|
||||
* the Windows MinGW CI job without the unmerged compat shim (#352). */
|
||||
#define main coli_glm_main_unused
|
||||
#include "../glm.c"
|
||||
#include "../colibri.c"
|
||||
#undef main
|
||||
|
||||
#include <math.h>
|
||||
|
||||
@@ -6,7 +6,7 @@
|
||||
#include <string.h>
|
||||
#include <unistd.h>
|
||||
#define main coli_glm_main_unused
|
||||
#include "../glm.c"
|
||||
#include "../colibri.c"
|
||||
#undef main
|
||||
|
||||
static int fail(const char *s){ fprintf(stderr,"FAIL: %s\n",s); return 1; }
|
||||
|
||||
+64
-57
@@ -9,6 +9,7 @@ import {
|
||||
Database,
|
||||
Feather,
|
||||
Gauge,
|
||||
Globe,
|
||||
HardDrive,
|
||||
KeyRound,
|
||||
Layers,
|
||||
@@ -34,6 +35,7 @@ import { Brain } from "./Brain"
|
||||
import { Profiling } from "./Profiling"
|
||||
import { persistPublicSettings, stored } from "@/lib/storage"
|
||||
import { cn } from "@/lib/utils"
|
||||
import { useLocale } from "./i18n"
|
||||
|
||||
const message = (role: ChatMessage["role"], content: string): ChatMessage => {
|
||||
let id: string
|
||||
@@ -42,15 +44,12 @@ const message = (role: ChatMessage["role"], content: string): ChatMessage => {
|
||||
}
|
||||
|
||||
export default function App() {
|
||||
// When the page is served by the engine itself (coli web), same-origin is the
|
||||
// right default: no CORS, no manual endpoint editing. The Vite dev server
|
||||
// (port 5173) keeps the classic default.
|
||||
const { t, locale, setLocale, locales } = useLocale()
|
||||
|
||||
const servedByEngine = typeof window !== "undefined" && window.location.port !== "5173" && window.location.protocol.startsWith("http")
|
||||
const defaultBase = servedByEngine ? `${window.location.origin}/v1` : "http://127.0.0.1:8000/v1"
|
||||
const [baseUrl, setBaseUrl] = useState(() => {
|
||||
const saved = stored(localStorage, "colibri.baseUrl", defaultBase)
|
||||
// migrate: a stored FACTORY default pointing at another origin would trip CORS
|
||||
// when the page is engine-served — upgrade it to same-origin once.
|
||||
if (servedByEngine && saved === "http://127.0.0.1:8000/v1" && defaultBase !== saved) return defaultBase
|
||||
return saved
|
||||
})
|
||||
@@ -120,7 +119,7 @@ export default function App() {
|
||||
const result = await getHealth(baseUrl, apiKey)
|
||||
if (!disposed) { setHealth(result); setHealthError("") }
|
||||
} catch (cause) {
|
||||
if (!disposed) setHealthError(cause instanceof Error ? cause.message : "Runtime metrics unavailable")
|
||||
if (!disposed) setHealthError(cause instanceof Error ? cause.message : "status.runtimeUnavailable")
|
||||
}
|
||||
}
|
||||
const timer = window.setInterval(() => void poll(), 5000)
|
||||
@@ -155,13 +154,13 @@ export default function App() {
|
||||
} catch (cause) {
|
||||
if (!controller.signal.aborted) {
|
||||
setHealth(null)
|
||||
setHealthError(cause instanceof Error ? cause.message : "Runtime metrics unavailable")
|
||||
setHealthError(cause instanceof Error ? cause.message : "status.runtimeUnavailable")
|
||||
}
|
||||
}
|
||||
} catch (cause) {
|
||||
if (controller.signal.aborted) return
|
||||
setConnected(false)
|
||||
setError(cause instanceof Error ? cause.message : "Could not reach the server.")
|
||||
setError(cause instanceof Error ? cause.message : "status.serverError")
|
||||
} finally {
|
||||
if (probeRef.current === controller) { probeRef.current = null; setConnecting(false) }
|
||||
}
|
||||
@@ -227,7 +226,7 @@ export default function App() {
|
||||
if (controller.signal.aborted) {
|
||||
updateMessages((current) => current.filter((item) => item.id !== assistant.id || item.content))
|
||||
} else {
|
||||
setError(cause instanceof Error ? cause.message : "Generation failed.")
|
||||
setError(cause instanceof Error ? cause.message : "status.generationFailed")
|
||||
updateMessages((current) => current.filter((item) => item.id !== assistant.id || item.content))
|
||||
}
|
||||
} finally {
|
||||
@@ -241,22 +240,22 @@ export default function App() {
|
||||
<aside className="sidebar">
|
||||
<div className="brand-row">
|
||||
<div className="brand-mark"><Feather className="size-5" /></div>
|
||||
<div><h1>colibrì</h1><p>local giant, tiny footprint</p></div>
|
||||
<div><h1>colibrì</h1><p>{t("brand.tagline")}</p></div>
|
||||
</div>
|
||||
|
||||
<section className="side-section">
|
||||
<div className="section-title"><Link2 className="size-3.5" /> Connection</div>
|
||||
<label>API endpoint<Input value={baseUrl} onChange={(event) => setBaseUrl(event.target.value)} /></label>
|
||||
<label>API key<div className="relative"><KeyRound className="field-icon" /><Input className="pl-9" type="password" value={apiKey} placeholder="optional" onChange={(event) => setApiKey(event.target.value)} /></div><span className="field-help">Kept in memory only · sent to this endpoint</span></label>
|
||||
<div className="section-title"><Link2 className="size-3.5" /> {t("sidebar.connection")}</div>
|
||||
<label>{t("sidebar.endpoint")}<Input value={baseUrl} onChange={(event) => setBaseUrl(event.target.value)} /></label>
|
||||
<label>{t("sidebar.apiKey")}<div className="relative"><KeyRound className="field-icon" /><Input className="pl-9" type="password" value={apiKey} placeholder={t("sidebar.apiKeyPlaceholder")} onChange={(event) => setApiKey(event.target.value)} /></div><span className="field-help">{t("sidebar.apiKeyHelp")}</span></label>
|
||||
<Button type="button" variant="secondary" onClick={connect} disabled={connecting}>
|
||||
{connecting ? <LoaderCircle className="size-4 animate-spin" /> : <RefreshCw className="size-4" />}
|
||||
Probe server
|
||||
{t("sidebar.probe")}
|
||||
</Button>
|
||||
<div className={cn("connection-state", connected && "connected")} aria-live="polite"><span />{connected ? "Engine reachable" : "Not connected"}</div>
|
||||
<div className={cn("connection-state", connected && "connected")} aria-live="polite"><span />{connected ? t("status.connected") : t("status.notConnected")}</div>
|
||||
</section>
|
||||
|
||||
<section className="side-section runtime-section" aria-live="polite">
|
||||
<div className="section-title"><Activity className="size-3.5" /> Runtime</div>
|
||||
<div className="section-title"><Activity className="size-3.5" /> {t("sidebar.runtime")}</div>
|
||||
{health?.hwinfo ? <div className="hw-panel">
|
||||
{health.hwinfo.cpu ? <div className="hw-row"><Cpu className="size-3.5" /><span>{health.hwinfo.cpu}</span></div> : null}
|
||||
{health.hwinfo.gpus > 0 ? <div className="hw-row"><MonitorDot className="size-3.5" /><span>{health.hwinfo.gpus}× GPU<small>{health.hwinfo.vram_total_gb.toFixed(0)} GB VRAM</small></span></div> : null}
|
||||
@@ -265,66 +264,74 @@ export default function App() {
|
||||
</div> : null}
|
||||
{health?.scheduler ? <>
|
||||
<div className="runtime-grid">
|
||||
<div><span>Active</span><strong>{active}<small> / {capacity}</small></strong></div>
|
||||
<div><span>Queued</span><strong>{health.scheduler.queued}<small> / {health.scheduler.max_queue}</small></strong></div>
|
||||
<div><span>Completed</span><strong>{health.scheduler.completed}</strong></div>
|
||||
<div><span>Failures</span><strong>{failures}</strong></div>
|
||||
<div><span>{t("dashboard.active")}</span><strong>{active}<small> / {capacity}</small></strong></div>
|
||||
<div><span>{t("dashboard.queued")}</span><strong>{health.scheduler.queued}<small> / {health.scheduler.max_queue}</small></strong></div>
|
||||
<div><span>{t("dashboard.completed")}</span><strong>{health.scheduler.completed}</strong></div>
|
||||
<div><span>{t("dashboard.failures")}</span><strong>{failures}</strong></div>
|
||||
</div>
|
||||
{health.tiers ? (() => {
|
||||
const t = health.tiers
|
||||
const total = Math.max(t.vram + t.ram + t.disk, 1)
|
||||
const ti = health.tiers
|
||||
const total = Math.max(ti.vram + ti.ram + ti.disk, 1)
|
||||
return <div className="tier-panel">
|
||||
<div className="tier-bar" role="img" aria-label={`Experts: ${t.vram} VRAM, ${t.ram} RAM, ${t.disk} disk`}>
|
||||
<span className="tier-vram" style={{ width: `${(100 * t.vram) / total}%` }} />
|
||||
<span className="tier-ram" style={{ width: `${(100 * t.ram) / total}%` }} />
|
||||
<span className="tier-disk" style={{ width: `${(100 * t.disk) / total}%` }} />
|
||||
<div className="tier-bar" role="img" aria-label={t("tier.ariaLabel", { vram: ti.vram, ram: ti.ram, disk: ti.disk })}>
|
||||
<span className="tier-vram" style={{ width: `${(100 * ti.vram) / total}%` }} />
|
||||
<span className="tier-ram" style={{ width: `${(100 * ti.ram) / total}%` }} />
|
||||
<span className="tier-disk" style={{ width: `${(100 * ti.disk) / total}%` }} />
|
||||
</div>
|
||||
<div className="tier-legend">
|
||||
<span><i className="tier-vram" />VRAM <strong>{t.vram.toLocaleString()}</strong><small>{t.vram_gb.toFixed(1)} GB</small></span>
|
||||
<span><i className="tier-ram" />RAM <strong>{t.ram.toLocaleString()}</strong><small>{t.ram_gb.toFixed(1)} GB</small></span>
|
||||
<span><i className="tier-disk" />Disk <strong>{t.disk.toLocaleString()}</strong></span>
|
||||
<span><i className="tier-vram" />{t("tier.vram")} <strong>{ti.vram.toLocaleString()}</strong><small>{ti.vram_gb.toFixed(1)} GB</small></span>
|
||||
<span><i className="tier-ram" />{t("tier.ram")} <strong>{ti.ram.toLocaleString()}</strong><small>{ti.ram_gb.toFixed(1)} GB</small></span>
|
||||
<span><i className="tier-disk" />{t("tier.disk")} <strong>{ti.disk.toLocaleString()}</strong></span>
|
||||
</div>
|
||||
</div>
|
||||
})() : null}
|
||||
{totalTokens.prompt + totalTokens.completion > 0 ? <div className="session-stats">
|
||||
<span><Database className="size-3" /> Session: <strong>{totalTokens.prompt.toLocaleString()}</strong> prompt + <strong>{totalTokens.completion.toLocaleString()}</strong> completion</span>
|
||||
<span><Database className="size-3" /> {t("dashboard.session")} <strong>{totalTokens.prompt.toLocaleString()}</strong> {t("dashboard.prompt")} + <strong>{totalTokens.completion.toLocaleString()}</strong> {t("dashboard.completion")}</span>
|
||||
</div> : null}
|
||||
<div className="runtime-foot"><span className="runtime-dot" /> Scheduler online <code>{kvSlots} KV</code></div>
|
||||
</> : <p className="runtime-unavailable">{connected ? (healthError || "Runtime metrics unavailable") : "Probe the server to inspect runtime state."}</p>}
|
||||
<div className="runtime-foot"><span className="runtime-dot" /> {t("sidebar.schedulerOnline")} <code>{kvSlots} KV</code></div>
|
||||
</> : <p className="runtime-unavailable">{connected ? (healthError ? t(healthError) : t("status.runtimeUnavailable")) : t("sidebar.runtimeProbe")}</p>}
|
||||
</section>
|
||||
|
||||
<section className="side-section">
|
||||
<div className="section-title"><SlidersHorizontal className="size-3.5" /> Inference</div>
|
||||
<label>Model<select value={model} onChange={(event) => setModel(event.target.value)}>{models.length ? models.map((id) => <option key={id}>{id}</option>) : <option>{model}</option>}</select></label>
|
||||
{health?.kv_slots && health.kv_slots > 1 ? <label>KV session<select value={cacheSlot} onChange={(event) => setCacheSlot(Number(event.target.value))} disabled={loading}>
|
||||
{Array.from({ length: kvSlots }, (_, slot) => <option key={slot} value={slot}>Session {slot + 1}</option>)}
|
||||
</select><span className="field-help">Isolated context · conversation follows the selected slot</span></label> : null}
|
||||
<label><span className="label-line"><span>Temperature</span><code>{temperature.toFixed(1)}</code></span><input className="range" type="range" min="0" max="2" step="0.1" value={temperature} onChange={(event) => setTemperature(Number(event.target.value))} /></label>
|
||||
<label>Max output tokens<Input type="number" min={1} max={4096} value={maxTokens} onChange={(event) => { const value = Number(event.target.value); if (Number.isFinite(value)) setMaxTokens(Math.min(4096, Math.max(1, Math.round(value)))) }} /></label>
|
||||
<div className="section-title"><SlidersHorizontal className="size-3.5" /> {t("sidebar.inference")}</div>
|
||||
<label>{t("sidebar.model")}<select value={model} onChange={(event) => setModel(event.target.value)}>{models.length ? models.map((id) => <option key={id}>{id}</option>) : <option>{model}</option>}</select></label>
|
||||
{health?.kv_slots && health.kv_slots > 1 ? <label>{t("sidebar.kvSession")}<select value={cacheSlot} onChange={(event) => setCacheSlot(Number(event.target.value))} disabled={loading}>
|
||||
{Array.from({ length: kvSlots }, (_, slot) => <option key={slot} value={slot}>{t("sidebar.sessionLabel", { slot: slot + 1 })}</option>)}
|
||||
</select><span className="field-help">{t("sidebar.kvSessionHelp")}</span></label> : null}
|
||||
<label><span className="label-line"><span>{t("sidebar.temperature")}</span><code>{temperature.toFixed(1)}</code></span><input className="range" type="range" min="0" max="2" step="0.1" value={temperature} onChange={(event) => setTemperature(Number(event.target.value))} /></label>
|
||||
<label>{t("sidebar.maxTokens")}<Input type="number" min={1} max={4096} value={maxTokens} onChange={(event) => { const value = Number(event.target.value); if (Number.isFinite(value)) setMaxTokens(Math.min(4096, Math.max(1, Math.round(value)))) }} /></label>
|
||||
<button type="button" className={cn("toggle-row", thinking && "active")} aria-pressed={thinking} onClick={() => setThinking((value) => !value)}>
|
||||
<span><BrainCircuit className="size-4" /> Reasoning</span><i><b /></i>
|
||||
<span><BrainCircuit className="size-4" /> {t("sidebar.reasoning")}</span><i><b /></i>
|
||||
</button>
|
||||
</section>
|
||||
|
||||
<div className="sidebar-foot"><Cpu className="size-3.5" /><span>OpenAI-compatible transport</span></div>
|
||||
<div className="sidebar-foot">
|
||||
<div><Cpu className="size-3.5" /><span>{t("sidebar.transport")}</span></div>
|
||||
<div className="locale-switcher">
|
||||
<Globe className="size-3.5" />
|
||||
<select value={locale} onChange={(e) => setLocale(e.target.value)}>
|
||||
{locales.map((l) => <option key={l.code} value={l.code}>{l.label}</option>)}
|
||||
</select>
|
||||
</div>
|
||||
</div>
|
||||
</aside>
|
||||
|
||||
<main className="chat-panel">
|
||||
<header className="topbar">
|
||||
<div><span className="eyebrow">ACTIVE MODEL</span><strong>{model}</strong></div>
|
||||
<div><span className="eyebrow">{t("topbar.activeModel")}</span><strong>{model}</strong></div>
|
||||
<div className="view-tabs">
|
||||
<button className={view === "chat" ? "active" : ""} onClick={() => setView("chat")}><MessageSquareText className="size-3.5" /> Chat</button>
|
||||
<button className={view === "brain" ? "active" : ""} onClick={() => setView("brain")}><BrainCircuit className="size-3.5" /> Brain</button>
|
||||
<button className={view === "profiling" ? "active" : ""} onClick={() => setView("profiling")}><Gauge className="size-3.5" /> Profiling</button>
|
||||
<button className={view === "chat" ? "active" : ""} onClick={() => setView("chat")}><MessageSquareText className="size-3.5" /> {t("nav.chat")}</button>
|
||||
<button className={view === "brain" ? "active" : ""} onClick={() => setView("brain")}><BrainCircuit className="size-3.5" /> {t("nav.brain")}</button>
|
||||
<button className={view === "profiling" ? "active" : ""} onClick={() => setView("profiling")}><Gauge className="size-3.5" /> {t("nav.profiling")}</button>
|
||||
</div>
|
||||
<div className="top-actions">
|
||||
{loading && tokenCount > 0 ? <Badge className="badge-live"><Zap className="size-3 flash" /> {tokenCount} tokens</Badge> : null}
|
||||
{!loading && tokPerSec != null ? <Badge className="badge-speed"><Gauge className="size-3" /> {tokPerSec.toFixed(1)} tok/s</Badge> : null}
|
||||
{loading && tokenCount > 0 ? <Badge className="badge-live"><Zap className="size-3 flash" /> {t("topbar.tokens", { n: tokenCount })}</Badge> : null}
|
||||
{!loading && tokPerSec != null ? <Badge className="badge-speed"><Gauge className="size-3" /> {t("topbar.tokPerSec", { n: tokPerSec.toFixed(1) })}</Badge> : null}
|
||||
{!loading && ttft != null ? <Badge><Timer className="size-3" /> TTFT {(ttft/1000).toFixed(1)}s</Badge> : null}
|
||||
{!loading && lastRun?.usage ? <Badge><Layers className="size-3" /> {lastRun.usage.prompt_tokens}→{lastRun.usage.completion_tokens}</Badge> : null}
|
||||
{lastRun?.queueWaitMs != null ? <Badge><Clock className="size-3" /> queue {Math.round(lastRun.queueWaitMs)}ms</Badge> : null}
|
||||
<Badge><MonitorDot className="size-3" /> slot {cacheSlot + 1}</Badge>
|
||||
<Button variant="ghost" size="sm" onClick={() => { updateMessages([]); setTokPerSec(null); setTtft(null); setTokenCount(0); setTotalTokens({prompt:0,completion:0}) }} disabled={!messages.length || loading}><Trash2 className="size-3.5" /> Clear</Button>
|
||||
<Badge><MonitorDot className="size-3" /> {t("topbar.slot", { n: cacheSlot + 1 })}</Badge>
|
||||
<Button variant="ghost" size="sm" onClick={() => { updateMessages([]); setTokPerSec(null); setTtft(null); setTokenCount(0); setTotalTokens({prompt:0,completion:0}) }} disabled={!messages.length || loading}><Trash2 className="size-3.5" /> {t("topbar.clear")}</Button>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
@@ -335,11 +342,11 @@ export default function App() {
|
||||
{!messages.length ? (
|
||||
<div className="empty-state">
|
||||
<div className="orb"><Feather /></div>
|
||||
<span className="eyebrow">COLIBRÌ ENGINE</span>
|
||||
<h2>Ask the giant.<br /><em>Keep the machine yours.</em></h2>
|
||||
<p>Connect to a local colibrì server and stream responses directly from your hardware. Nothing leaves the endpoint you choose.</p>
|
||||
<span className="eyebrow">{t("hero.title")}</span>
|
||||
<h2>{t("hero.subtitle")}<br /><em>{t("hero.tagline")}</em></h2>
|
||||
<p>{t("hero.description")}</p>
|
||||
<div className="suggestions">
|
||||
{["Explain how expert routing works", "Write a small C benchmark", "Compare RAM and VRAM caching"].map((item) => <button key={item} onClick={() => setDraft(item)}>{item}<ArrowUp className="size-3.5 rotate-45" /></button>)}
|
||||
{[t("prompts.routing"), t("prompts.benchmark"), t("prompts.caching")].map((item) => <button key={item} onClick={() => setDraft(item)}>{item}<ArrowUp className="size-3.5 rotate-45" /></button>)}
|
||||
</div>
|
||||
</div>
|
||||
) : (
|
||||
@@ -347,7 +354,7 @@ export default function App() {
|
||||
{messages.map((item) => (
|
||||
<article key={item.id} className={cn("message", item.role)}>
|
||||
<div className="avatar">{item.role === "user" ? "Y" : <Feather className="size-4" />}</div>
|
||||
<div><div className="message-meta">{item.role === "user" ? "You" : "colibrì"}</div><div className="message-body">{item.content || <span className="typing" aria-label="Generating"><i /><i /><i /></span>}</div></div>
|
||||
<div><div className="message-meta">{item.role === "user" ? t("chat.you") : t("chat.colibri")}</div><div className="message-body">{item.content || <span className="typing" aria-label="Generating"><i /><i /><i /></span>}</div></div>
|
||||
</article>
|
||||
))}
|
||||
<div ref={bottomRef} />
|
||||
@@ -356,10 +363,10 @@ export default function App() {
|
||||
</div>
|
||||
|
||||
<div className="composer-wrap">
|
||||
{error && <div className="error-banner" role="alert">{error}</div>}
|
||||
{error && <div className="error-banner" role="alert">{t(error)}</div>}
|
||||
<div className="composer">
|
||||
<Textarea value={draft} onChange={(event) => setDraft(event.target.value)} placeholder="Message colibrì…" onKeyDown={(event) => { if (event.key === "Enter" && !event.shiftKey && !event.nativeEvent.isComposing) { event.preventDefault(); void send() } }} />
|
||||
<div className="composer-foot"><span><MessageSquareText className="size-3.5" /> Enter to send · Shift+Enter for newline</span>{loading ? <Button variant="destructive" size="icon" aria-label="Stop generation" onClick={() => abortRef.current?.abort()}><CircleStop className="size-4" /></Button> : <Button size="icon" aria-label="Send message" disabled={!canSend} onClick={() => void send()}><ArrowUp className="size-4" /></Button>}</div>
|
||||
<Textarea value={draft} onChange={(event) => setDraft(event.target.value)} placeholder={t("chat.placeholder")} onKeyDown={(event) => { if (event.key === "Enter" && !event.shiftKey && !event.nativeEvent.isComposing) { event.preventDefault(); void send() } }} />
|
||||
<div className="composer-foot"><span><MessageSquareText className="size-3.5" /> {t("chat.inputHint")}</span>{loading ? <Button variant="destructive" size="icon" aria-label={t("chat.stop")} onClick={() => abortRef.current?.abort()}><CircleStop className="size-4" /></Button> : <Button size="icon" aria-label={t("chat.send")} disabled={!canSend} onClick={() => void send()}><ArrowUp className="size-4" /></Button>}</div>
|
||||
</div>
|
||||
</div>
|
||||
</>}
|
||||
|
||||
+21
-22
@@ -2,27 +2,26 @@ import { useEffect, useRef, useState } from "react"
|
||||
import { BrainCircuit, Flame, Layers } from "lucide-react"
|
||||
|
||||
import { endpoint } from "@/lib/api"
|
||||
import { useLocale } from "./i18n"
|
||||
|
||||
interface ExpertMap { rows: number; cols: number; map: string; hits: string; seq: number }
|
||||
interface AtlasEntry { affinity: Record<string, number>; entropy: number; top: string; label: string }
|
||||
|
||||
const TIER_NAME = ["Disk", "RAM", "VRAM"]
|
||||
const TIER_KEYS = ["tier.disk", "tier.ram", "tier.vram"] as const
|
||||
const TIER_RGB: [number, number, number][] = [[58, 71, 80], [90, 155, 216], [78, 214, 165]]
|
||||
|
||||
/* Layer-depth heuristic: what this region of the network tends to specialise in.
|
||||
* Honest framing — these are the depth roles observed across MoE interpretability
|
||||
* work, not per-expert ground truth (that needs co-activation analysis, #119). */
|
||||
function depthRole(row: number, rows: number, isMtp: boolean): string {
|
||||
if (isMtp) return "MTP head — drafts the next token for speculative decoding"
|
||||
function depthRoleKey(row: number, rows: number, isMtp: boolean): string {
|
||||
if (isMtp) return "brain.mtp"
|
||||
const f = row / Math.max(rows - 1, 1)
|
||||
if (f < 0.2) return "early layers — surface features: tokens, spelling, local syntax"
|
||||
if (f < 0.45) return "lower-middle — phrase structure, word relations, simple facts"
|
||||
if (f < 0.7) return "upper-middle — semantics, long-range context, reasoning steps"
|
||||
if (f < 0.9) return "late layers — planning the answer, style, coherence"
|
||||
return "final layers — output shaping: picks the actual next-token distribution"
|
||||
if (f < 0.2) return "brain.early"
|
||||
if (f < 0.45) return "brain.lowerMiddle"
|
||||
if (f < 0.7) return "brain.upperMiddle"
|
||||
if (f < 0.9) return "brain.late"
|
||||
return "brain.final"
|
||||
}
|
||||
|
||||
export function Brain({ baseUrl, apiKey, connected }: { baseUrl: string; apiKey: string; connected: boolean }) {
|
||||
const { t } = useLocale()
|
||||
const canvasRef = useRef<HTMLCanvasElement>(null)
|
||||
const wrapRef = useRef<HTMLDivElement>(null)
|
||||
const [wrapSize, setWrapSize] = useState({ w: 1200, h: 700 })
|
||||
@@ -142,18 +141,18 @@ export function Brain({ baseUrl, apiKey, connected }: { baseUrl: string; apiKey:
|
||||
return (
|
||||
<div className="brain-page">
|
||||
<div className="brain-head">
|
||||
<div className="section-title"><BrainCircuit className="size-4" /> Expert Cortex — {data ? `${data.rows} layers × ${data.cols} experts` : "waiting for engine"}</div>
|
||||
<div className="section-title"><BrainCircuit className="size-4" /> {t("brain.title")} — {data ? t("brain.layers", { rows: data.rows, cols: data.cols }) : t("brain.waiting")}</div>
|
||||
<div className="brain-legend">
|
||||
<span><i style={{ background: "#4ed6a5" }} /> VRAM {totals[2].toLocaleString()}</span>
|
||||
<span><i style={{ background: "#5a9bd8" }} /> RAM {totals[1].toLocaleString()}</span>
|
||||
<span><i style={{ background: "#3a4750" }} /> Disk {totals[0].toLocaleString()}</span>
|
||||
<span><Flame className="size-3" /> brightness = routing heat</span>
|
||||
<span className="brain-pulse-hint">⚡ white flash = routed this turn</span>
|
||||
<span><i style={{ background: "#4ed6a5" }} /> {t("tier.vram")} {totals[2].toLocaleString()}</span>
|
||||
<span><i style={{ background: "#5a9bd8" }} /> {t("tier.ram")} {totals[1].toLocaleString()}</span>
|
||||
<span><i style={{ background: "#3a4750" }} /> {t("tier.disk")} {totals[0].toLocaleString()}</span>
|
||||
<span><Flame className="size-3" /> {t("brain.brightnessHint")}</span>
|
||||
<span className="brain-pulse-hint">{t("brain.flashHint")}</span>
|
||||
</div>
|
||||
</div>
|
||||
<div className="brain-canvas-wrap" ref={wrapRef}>
|
||||
<canvas ref={canvasRef} onMouseMove={onMove} onMouseLeave={() => setTip(null)} />
|
||||
{!connected && <p className="runtime-unavailable">Connect to the engine to see the cortex.</p>}
|
||||
{!connected && <p className="runtime-unavailable">{t("brain.connectHint")}</p>}
|
||||
</div>
|
||||
{tip && data && (() => {
|
||||
const isMtp = tip.row === data.rows - 1
|
||||
@@ -162,16 +161,16 @@ export function Brain({ baseUrl, apiKey, connected }: { baseUrl: string; apiKey:
|
||||
return (
|
||||
<div className="brain-tip" style={{ left: tip.x + 14, top: tip.y + 14 }}>
|
||||
<div className="brain-tip-title"><Layers className="size-3" /> Layer {realLayer}{isMtp ? " (MTP)" : ""} · Expert {tip.col}</div>
|
||||
<div>Tier: <strong style={{ color: ["#8b9aa3", "#5a9bd8", "#4ed6a5"][tip.tier] }}>{TIER_NAME[tip.tier]}</strong></div>
|
||||
<div>Heat: <strong>{tip.heat === 0 ? "never routed" : `~2^${tip.heat} selections`}</strong></div>
|
||||
<div>Tier: <strong style={{ color: ["#8b9aa3", "#5a9bd8", "#4ed6a5"][tip.tier] }}>{t(TIER_KEYS[tip.tier])}</strong></div>
|
||||
<div>Heat: <strong>{tip.heat === 0 ? t("brain.neverRouted") : t("brain.selections", { heat: tip.heat })}</strong></div>
|
||||
{entry ? <>
|
||||
<div className={entry.label.startsWith("specialist") ? "brain-tip-spec" : undefined}>
|
||||
{entry.label.startsWith("specialist") ? `⭐ Specialist: ${entry.top}` : "Generalist"}
|
||||
{entry.label.startsWith("specialist") ? t("brain.specialist", { top: entry.top }) : t("brain.generalist")}
|
||||
<small> (entropy {entry.entropy})</small>
|
||||
</div>
|
||||
<div className="brain-tip-aff">{Object.entries(entry.affinity).sort((a, b) => b[1] - a[1]).slice(0, 3)
|
||||
.map(([c, p]) => `${c} ${Math.round(p * 100)}%`).join(" · ")}</div>
|
||||
</> : <div className="brain-tip-role">{depthRole(tip.row, data.rows, isMtp)}</div>}
|
||||
</> : <div className="brain-tip-role">{t(depthRoleKey(tip.row, data.rows, isMtp))}</div>}
|
||||
</div>
|
||||
)
|
||||
})()}
|
||||
|
||||
@@ -1,4 +1,18 @@
|
||||
import { Component, type ReactNode } from "react"
|
||||
import { useLocale } from "./i18n"
|
||||
|
||||
function ErrorFallback({ error, onRetry }: { error: Error; onRetry: () => void }) {
|
||||
const { t } = useLocale()
|
||||
return (
|
||||
<div style={{ padding: "2rem", fontFamily: "ui-monospace, monospace", color: "#e5e7eb", background: "#0b0f10", minHeight: "100vh" }}>
|
||||
<h2 style={{ color: "#4ed6a5" }}>{t("error.title")}</h2>
|
||||
<p style={{ color: "#9ca3af" }}>{t("error.hint")}</p>
|
||||
<pre style={{ whiteSpace: "pre-wrap", color: "#f87171" }}>{String(error)}</pre>
|
||||
<button onClick={onRetry} style={{ marginTop: "1rem", padding: "0.5rem 1rem", background: "#1f2937", color: "#e5e7eb", border: "1px solid #374151", borderRadius: 8, cursor: "pointer" }}>{t("error.retry")}</button>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
interface State { error: Error | null; stack: string }
|
||||
export class ErrorBoundary extends Component<{ children: ReactNode }, State> {
|
||||
state: State = { error: null, stack: "" }
|
||||
@@ -9,11 +23,6 @@ export class ErrorBoundary extends Component<{ children: ReactNode }, State> {
|
||||
}
|
||||
render() {
|
||||
if (!this.state.error) return this.props.children
|
||||
return <div style={{ padding: "2rem", fontFamily: "ui-monospace, monospace", color: "#e5e7eb", background: "#0b0f10", minHeight: "100vh" }}>
|
||||
<h2 style={{ color: "#4ed6a5" }}>colibrì UI hit an error</h2>
|
||||
<p style={{ color: "#9ca3af" }}>The engine is unaffected. Try refreshing.</p>
|
||||
<pre style={{ whiteSpace: "pre-wrap", color: "#f87171" }}>{String(this.state.error)}</pre>
|
||||
<button onClick={() => this.setState({ error: null, stack: "" })} style={{ marginTop: "1rem", padding: "0.5rem 1rem", background: "#1f2937", color: "#e5e7eb", border: "1px solid #374151", borderRadius: 8, cursor: "pointer" }}>Retry</button>
|
||||
</div>
|
||||
return <ErrorFallback error={this.state.error} onRetry={() => this.setState({ error: null, stack: "" })} />
|
||||
}
|
||||
}
|
||||
|
||||
+26
-32
@@ -2,19 +2,14 @@ import { useEffect, useState } from "react"
|
||||
import { Activity, Gauge, HardDrive, Timer } from "lucide-react"
|
||||
|
||||
import { getProfile, type ProfileTurn } from "@/lib/api"
|
||||
import { useLocale } from "./i18n"
|
||||
|
||||
/* Wall-time phases stacked per turn. The order is the palette's CVD-safe slot
|
||||
* order (validated as a set on this surface) — identity never leans on colour
|
||||
* alone: segments keep 2px gaps, the legend is always shown and the table
|
||||
* carries the exact numbers. Disk *service* time is reported separately: it
|
||||
* runs on I/O threads overlapped with compute, so only the stall the compute
|
||||
* thread actually felt (I/O wait) belongs inside the wall-time stack. */
|
||||
const PHASES = [
|
||||
{ key: "expert_wait_s", name: "I/O wait", color: "#3987e5" },
|
||||
{ key: "expert_matmul_s", name: "Expert matmul", color: "#199e70" },
|
||||
{ key: "attention_s", name: "Attention", color: "#c98500" },
|
||||
{ key: "lm_head_s", name: "LM head", color: "#008300" },
|
||||
{ key: "other_s", name: "Other", color: "#9085e9" },
|
||||
{ key: "expert_wait_s", i18n: "profile.ioWait", color: "#3987e5" },
|
||||
{ key: "expert_matmul_s", i18n: "profile.expertMatmul", color: "#199e70" },
|
||||
{ key: "attention_s", i18n: "profile.attention", color: "#c98500" },
|
||||
{ key: "lm_head_s", i18n: "profile.lmHead", color: "#008300" },
|
||||
{ key: "other_s", i18n: "profile.other", color: "#9085e9" },
|
||||
] as const
|
||||
|
||||
interface Turn extends ProfileTurn { other_s: number; toks: number }
|
||||
@@ -28,8 +23,9 @@ const derive = (turn: ProfileTurn): Turn => ({
|
||||
const seconds = (value: number) => (value >= 10 ? value.toFixed(1) : value.toFixed(2)) + "s"
|
||||
|
||||
function ShareBar({ label, turns }: { label: string; turns: Turn[] }) {
|
||||
const { t } = useLocale()
|
||||
const total = turns.reduce((sum, turn) => sum + turn.wall_s, 0)
|
||||
const parts = PHASES.map((phase) => ({ ...phase, value: turns.reduce((sum, turn) => sum + turn[phase.key], 0) }))
|
||||
const parts = PHASES.map((phase) => ({ ...phase, name: t(phase.i18n), value: turns.reduce((sum, turn) => sum + turn[phase.key], 0) }))
|
||||
return (
|
||||
<div className="prof-share">
|
||||
<div className="prof-share-head"><span>{label}</span><code>{seconds(total)}</code></div>
|
||||
@@ -47,9 +43,7 @@ function ShareBar({ label, turns }: { label: string; turns: Turn[] }) {
|
||||
)
|
||||
}
|
||||
|
||||
/* Column chart over the recent turns; oldest on the left. Stacked mode draws the
|
||||
* wall-time composition, plain mode a single series (no legend — the title names it). */
|
||||
function TurnColumns({ turns, stacked, height, format }: { turns: Turn[]; stacked: boolean; height: number; format: (turn: Turn) => string }) {
|
||||
function TurnColumns({ turns, stacked, height, format, footLabel, footLabelOne }: { turns: Turn[]; stacked: boolean; height: number; format: (turn: Turn) => string; footLabel: string; footLabelOne: string }) {
|
||||
const [hover, setHover] = useState<number | null>(null)
|
||||
const peak = Math.max(...turns.map((turn) => (stacked ? turn.wall_s : turn.toks)), 1e-9)
|
||||
const gap = 2
|
||||
@@ -71,11 +65,10 @@ function TurnColumns({ turns, stacked, height, format }: { turns: Turn[]; stacke
|
||||
return h > 0.1 ? <rect key={`${index}-${phase.key}`} x={x} y={y + 0.35} width={width} height={Math.max(h - 0.7, 0.35)} fill={phase.color} opacity={hover === null || hover === index ? 1 : 0.45} /> : null
|
||||
})
|
||||
})}
|
||||
{/* hit targets bigger than the marks */}
|
||||
{turns.map((_, index) => <rect key={index} x={index * (width + gap) - gap / 2} y="0" width={width + gap} height={height} fill="transparent" onMouseEnter={() => setHover(index)} />)}
|
||||
</svg>
|
||||
<div className="prof-plot-foot">
|
||||
<span>{turns.length > 1 ? `${turns.length} turns · oldest → newest` : "1 turn"}</span>
|
||||
<span>{turns.length > 1 ? footLabel : footLabelOne}</span>
|
||||
<code>{hover !== null && turns[hover] ? format(turns[hover]) : `peak ${stacked ? seconds(peak) : peak.toFixed(1) + " tok/s"}`}</code>
|
||||
</div>
|
||||
</div>
|
||||
@@ -83,6 +76,7 @@ function TurnColumns({ turns, stacked, height, format }: { turns: Turn[]; stacke
|
||||
}
|
||||
|
||||
export function Profiling({ baseUrl, apiKey, connected }: { baseUrl: string; apiKey: string; connected: boolean }) {
|
||||
const { t } = useLocale()
|
||||
const [turns, setTurns] = useState<Turn[]>([])
|
||||
|
||||
useEffect(() => {
|
||||
@@ -107,42 +101,42 @@ export function Profiling({ baseUrl, apiKey, connected }: { baseUrl: string; api
|
||||
return (
|
||||
<div className="prof-page">
|
||||
<div className="prof-head">
|
||||
<div className="section-title"><Gauge className="size-4" /> Profiling — where the engine spends each turn</div>
|
||||
<div className="section-title"><Gauge className="size-4" /> {t("profile.title")}</div>
|
||||
<div className="prof-legend">
|
||||
{PHASES.map((phase) => <span key={phase.key}><i style={{ background: phase.color }} />{phase.name}</span>)}
|
||||
{PHASES.map((phase) => <span key={phase.key}><i style={{ background: phase.color }} />{t(phase.i18n)}</span>)}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{!latest ? (
|
||||
<p className="runtime-unavailable">{connected ? "No profiled turns yet — send a chat message and the breakdown appears here." : "Connect to the engine to collect per-turn timings."}</p>
|
||||
<p className="runtime-unavailable">{connected ? t("profile.empty") : t("profile.connectHint")}</p>
|
||||
) : (
|
||||
<>
|
||||
<div className="prof-tiles">
|
||||
<div><span><Gauge className="size-3" /> Last turn</span><strong>{latest.toks.toFixed(1)}</strong><small>tok/s</small></div>
|
||||
<div><span><Timer className="size-3" /> Wall time</span><strong>{seconds(latest.wall_s)}</strong><small>{latest.prompt_tokens} → {latest.completion_tokens} tokens</small></div>
|
||||
<div><span><Activity className="size-3" /> Batching</span><strong>{latest.forwards > 0 ? (latest.completion_tokens / latest.forwards).toFixed(2) : "—"}</strong><small>tokens / forward</small></div>
|
||||
<div><span><HardDrive className="size-3" /> Disk service</span><strong>{seconds(latest.expert_disk_s)}</strong><small>overlapped with compute</small></div>
|
||||
<div><span><Gauge className="size-3" /> {t("profile.lastTurn")}</span><strong>{latest.toks.toFixed(1)}</strong><small>tok/s</small></div>
|
||||
<div><span><Timer className="size-3" /> {t("profile.wallTime")}</span><strong>{seconds(latest.wall_s)}</strong><small>{latest.prompt_tokens} → {latest.completion_tokens} tokens</small></div>
|
||||
<div><span><Activity className="size-3" /> {t("profile.batching")}</span><strong>{latest.forwards > 0 ? (latest.completion_tokens / latest.forwards).toFixed(2) : "—"}</strong><small>{t("profile.tokensPerForward")}</small></div>
|
||||
<div><span><HardDrive className="size-3" /> {t("profile.diskService")}</span><strong>{seconds(latest.expert_disk_s)}</strong><small>{t("profile.overlapped")}</small></div>
|
||||
</div>
|
||||
|
||||
<div className="prof-shares">
|
||||
<ShareBar label="Last turn" turns={[latest]} />
|
||||
{turns.length > 1 ? <ShareBar label={`Window · last ${turns.length} turns`} turns={turns} /> : null}
|
||||
<ShareBar label={t("profile.lastTurn")} turns={[latest]} />
|
||||
{turns.length > 1 ? <ShareBar label={t("profile.window", { n: turns.length })} turns={turns} /> : null}
|
||||
</div>
|
||||
|
||||
<div className="prof-charts">
|
||||
<div className="prof-chart">
|
||||
<div className="prof-chart-title">Throughput per turn (tok/s)</div>
|
||||
<TurnColumns turns={recent} stacked={false} height={36} format={(turn) => `${turn.toks.toFixed(1)} tok/s · ${turn.completion_tokens} tokens`} />
|
||||
<div className="prof-chart-title">{t("profile.throughputTitle")}</div>
|
||||
<TurnColumns turns={recent} stacked={false} height={36} footLabel={t("profile.turnsLabel", { n: recent.length })} footLabelOne={t("profile.oneTurn")} format={(turn) => `${turn.toks.toFixed(1)} tok/s · ${turn.completion_tokens} tokens`} />
|
||||
</div>
|
||||
<div className="prof-chart">
|
||||
<div className="prof-chart-title">Turn wall time by phase (s)</div>
|
||||
<TurnColumns turns={recent} stacked height={36} format={(turn) => `${seconds(turn.wall_s)} · ${PHASES.map((phase) => `${phase.name} ${seconds(turn[phase.key])}`).join(" · ")}`} />
|
||||
<div className="prof-chart-title">{t("profile.phaseTitle")}</div>
|
||||
<TurnColumns turns={recent} stacked height={36} footLabel={t("profile.turnsLabel", { n: recent.length })} footLabelOne={t("profile.oneTurn")} format={(turn) => `${seconds(turn.wall_s)} · ${PHASES.map((phase) => `${t(phase.i18n)} ${seconds(turn[phase.key])}`).join(" · ")}`} />
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div className="prof-table-wrap">
|
||||
<table className="prof-table">
|
||||
<thead><tr><th>Turn</th><th>Tokens</th><th>tok/s</th><th>Wall</th>{PHASES.map((phase) => <th key={phase.key}><i style={{ background: phase.color }} />{phase.name}</th>)}<th>Disk service</th></tr></thead>
|
||||
<thead><tr><th>{t("profile.turnCol")}</th><th>{t("profile.tokensCol")}</th><th>tok/s</th><th>{t("profile.wallCol")}</th>{PHASES.map((phase) => <th key={phase.key}><i style={{ background: phase.color }} />{t(phase.i18n)}</th>)}<th>{t("profile.diskService")}</th></tr></thead>
|
||||
<tbody>
|
||||
{recent.slice().reverse().map((turn, index) => (
|
||||
<tr key={turns.length - index}>
|
||||
@@ -156,7 +150,7 @@ export function Profiling({ baseUrl, apiKey, connected }: { baseUrl: string; api
|
||||
))}
|
||||
</tbody>
|
||||
</table>
|
||||
{diskService > 0 ? <p className="prof-note">Disk service is time spent reading experts on I/O threads; it overlaps with compute, so only the <em>I/O wait</em> the compute thread felt counts inside the wall-time stack. With multiple KV sessions the shares describe the whole engine over the turn's window.</p> : null}
|
||||
{diskService > 0 ? <p className="prof-note">{t("profile.diskNote")}</p> : null}
|
||||
</div>
|
||||
</>
|
||||
)}
|
||||
|
||||
@@ -0,0 +1,125 @@
|
||||
const en: Record<string, string> = {
|
||||
// nav
|
||||
"nav.chat": "Chat",
|
||||
"nav.brain": "Brain",
|
||||
"nav.profiling": "Profiling",
|
||||
|
||||
// brand
|
||||
"brand.tagline": "local giant, tiny footprint",
|
||||
|
||||
// sidebar — connection
|
||||
"sidebar.connection": "Connection",
|
||||
"sidebar.endpoint": "API endpoint",
|
||||
"sidebar.apiKey": "API key",
|
||||
"sidebar.apiKeyPlaceholder": "optional",
|
||||
"sidebar.apiKeyHelp": "Kept in memory only · sent to this endpoint",
|
||||
"sidebar.probe": "Probe server",
|
||||
"status.connected": "Engine reachable",
|
||||
"status.notConnected": "Not connected",
|
||||
"status.runtimeUnavailable": "Runtime metrics unavailable",
|
||||
"status.serverError": "Could not reach the server.",
|
||||
"status.generationFailed": "Generation failed.",
|
||||
|
||||
// sidebar — runtime
|
||||
"sidebar.runtime": "Runtime",
|
||||
"sidebar.runtimeProbe": "Probe the server to inspect runtime state.",
|
||||
"sidebar.schedulerOnline": "Scheduler online",
|
||||
"dashboard.active": "Active",
|
||||
"dashboard.queued": "Queued",
|
||||
"dashboard.completed": "Completed",
|
||||
"dashboard.failures": "Failures",
|
||||
"dashboard.session": "Session:",
|
||||
"dashboard.prompt": "prompt",
|
||||
"dashboard.completion": "completion",
|
||||
|
||||
// sidebar — tiers
|
||||
"tier.vram": "VRAM",
|
||||
"tier.ram": "RAM",
|
||||
"tier.disk": "Disk",
|
||||
"tier.ariaLabel": "Experts: {{vram}} VRAM, {{ram}} RAM, {{disk}} disk",
|
||||
|
||||
// sidebar — inference
|
||||
"sidebar.inference": "Inference",
|
||||
"sidebar.model": "Model",
|
||||
"sidebar.kvSession": "KV session",
|
||||
"sidebar.kvSessionHelp": "Isolated context · conversation follows the selected slot",
|
||||
"sidebar.sessionLabel": "Session {{slot}}",
|
||||
"sidebar.temperature": "Temperature",
|
||||
"sidebar.maxTokens": "Max output tokens",
|
||||
"sidebar.reasoning": "Reasoning",
|
||||
"sidebar.transport": "OpenAI-compatible transport",
|
||||
|
||||
// top bar
|
||||
"topbar.activeModel": "ACTIVE MODEL",
|
||||
"topbar.tokens": "{{n}} tokens",
|
||||
"topbar.tokPerSec": "{{n}} tok/s",
|
||||
"topbar.slot": "slot {{n}}",
|
||||
"topbar.clear": "Clear",
|
||||
|
||||
// hero / empty state
|
||||
"hero.title": "COLIBRÌ ENGINE",
|
||||
"hero.subtitle": "Ask the giant.",
|
||||
"hero.tagline": "Keep the machine yours.",
|
||||
"hero.description": "Connect to a local colibrì server and stream responses directly from your hardware. Nothing leaves the endpoint you choose.",
|
||||
"prompts.routing": "Explain how expert routing works",
|
||||
"prompts.benchmark": "Write a small C benchmark",
|
||||
"prompts.caching": "Compare RAM and VRAM caching",
|
||||
|
||||
// chat
|
||||
"chat.you": "You",
|
||||
"chat.colibri": "colibrì",
|
||||
"chat.placeholder": "Message colibrì…",
|
||||
"chat.inputHint": "Enter to send · Shift+Enter for newline",
|
||||
"chat.stop": "Stop generation",
|
||||
"chat.send": "Send message",
|
||||
|
||||
// brain
|
||||
"brain.title": "Expert Cortex",
|
||||
"brain.waiting": "waiting for engine",
|
||||
"brain.layers": "{{rows}} layers × {{cols}} experts",
|
||||
"brain.brightnessHint": "brightness = routing heat",
|
||||
"brain.flashHint": "⚡ white flash = routed this turn",
|
||||
"brain.connectHint": "Connect to the engine to see the cortex.",
|
||||
"brain.neverRouted": "never routed",
|
||||
"brain.selections": "~2^{{heat}} selections",
|
||||
"brain.specialist": "⭐ Specialist: {{top}}",
|
||||
"brain.generalist": "Generalist",
|
||||
"brain.mtp": "MTP head — drafts the next token for speculative decoding",
|
||||
"brain.early": "early layers — surface features: tokens, spelling, local syntax",
|
||||
"brain.lowerMiddle": "lower-middle — phrase structure, word relations, simple facts",
|
||||
"brain.upperMiddle": "upper-middle — semantics, long-range context, reasoning steps",
|
||||
"brain.late": "late layers — planning the answer, style, coherence",
|
||||
"brain.final": "final layers — output shaping: picks the actual next-token distribution",
|
||||
|
||||
// profiling
|
||||
"profile.title": "Profiling — where the engine spends each turn",
|
||||
"profile.ioWait": "I/O wait",
|
||||
"profile.expertMatmul": "Expert matmul",
|
||||
"profile.attention": "Attention",
|
||||
"profile.lmHead": "LM head",
|
||||
"profile.other": "Other",
|
||||
"profile.empty": "No profiled turns yet — send a chat message and the breakdown appears here.",
|
||||
"profile.connectHint": "Connect to the engine to collect per-turn timings.",
|
||||
"profile.lastTurn": "Last turn",
|
||||
"profile.wallTime": "Wall time",
|
||||
"profile.batching": "Batching",
|
||||
"profile.tokensPerForward": "tokens / forward",
|
||||
"profile.diskService": "Disk service",
|
||||
"profile.overlapped": "overlapped with compute",
|
||||
"profile.window": "Window · last {{n}} turns",
|
||||
"profile.throughputTitle": "Throughput per turn (tok/s)",
|
||||
"profile.phaseTitle": "Turn wall time by phase (s)",
|
||||
"profile.turnCol": "Turn",
|
||||
"profile.tokensCol": "Tokens",
|
||||
"profile.wallCol": "Wall",
|
||||
"profile.turnsLabel": "{{n}} turns · oldest → newest",
|
||||
"profile.oneTurn": "1 turn",
|
||||
"profile.diskNote": "Disk service is time spent reading experts on I/O threads; it overlaps with compute, so only the I/O wait the compute thread felt counts inside the wall-time stack. With multiple KV sessions the shares describe the whole engine over the turn's window.",
|
||||
|
||||
// error boundary
|
||||
"error.title": "colibrì UI hit an error",
|
||||
"error.hint": "The engine is unaffected. Try refreshing.",
|
||||
"error.retry": "Retry",
|
||||
}
|
||||
|
||||
export default en
|
||||
@@ -0,0 +1,78 @@
|
||||
import { createContext, useContext, useState, useCallback, useMemo, type ReactNode } from "react"
|
||||
import { createElement } from "react"
|
||||
import en from "./en"
|
||||
import zhCN from "./zh-CN"
|
||||
import zhTW from "./zh-TW"
|
||||
import it from "./it"
|
||||
|
||||
const LOCALES = [
|
||||
{ code: "en", label: "English" },
|
||||
{ code: "zh-CN", label: "简体中文" },
|
||||
{ code: "zh-TW", label: "繁體中文" },
|
||||
{ code: "it", label: "Italiano" },
|
||||
] as const
|
||||
|
||||
const DICTS: Record<string, Record<string, string>> = {
|
||||
"en": en,
|
||||
"zh-CN": zhCN,
|
||||
"zh-TW": zhTW,
|
||||
"it": it,
|
||||
}
|
||||
|
||||
const STORAGE_KEY = "colibri-locale"
|
||||
|
||||
function detectLocale(): string {
|
||||
try {
|
||||
const saved = localStorage.getItem(STORAGE_KEY)
|
||||
if (saved && DICTS[saved]) return saved
|
||||
} catch {}
|
||||
const nav = navigator.language || ""
|
||||
if (DICTS[nav]) return nav
|
||||
const prefix = nav.split("-")[0]
|
||||
if (prefix === "zh") return nav.includes("TW") || nav.includes("Hant") ? "zh-TW" : "zh-CN"
|
||||
for (const { code } of LOCALES) if (code.startsWith(prefix)) return code
|
||||
return "en"
|
||||
}
|
||||
|
||||
function interpolate(template: string, vars?: Record<string, string | number>): string {
|
||||
if (!vars) return template
|
||||
return template.replace(/\{\{(\w+)\}\}/g, (_, key) => String(vars[key] ?? `{{${key}}}`))
|
||||
}
|
||||
|
||||
interface LocaleContext {
|
||||
locale: string
|
||||
setLocale: (code: string) => void
|
||||
t: (key: string, vars?: Record<string, string | number>) => string
|
||||
locales: readonly { code: string; label: string }[]
|
||||
}
|
||||
|
||||
const Ctx = createContext<LocaleContext>({
|
||||
locale: "en",
|
||||
setLocale: () => {},
|
||||
t: (key) => key,
|
||||
locales: LOCALES,
|
||||
})
|
||||
|
||||
export function LocaleProvider({ children }: { children: ReactNode }) {
|
||||
const [locale, setLocaleState] = useState(detectLocale)
|
||||
|
||||
const setLocale = useCallback((code: string) => {
|
||||
if (!DICTS[code]) return
|
||||
setLocaleState(code)
|
||||
try { localStorage.setItem(STORAGE_KEY, code) } catch {}
|
||||
}, [])
|
||||
|
||||
const t = useCallback((key: string, vars?: Record<string, string | number>) => {
|
||||
const dict = DICTS[locale] || en
|
||||
const template = dict[key] ?? en[key] ?? key
|
||||
return interpolate(template, vars)
|
||||
}, [locale])
|
||||
|
||||
const value = useMemo(() => ({ locale, setLocale, t, locales: LOCALES }), [locale, setLocale, t])
|
||||
|
||||
return createElement(Ctx.Provider, { value }, children)
|
||||
}
|
||||
|
||||
export function useLocale() {
|
||||
return useContext(Ctx)
|
||||
}
|
||||
@@ -0,0 +1,113 @@
|
||||
const it: Record<string, string> = {
|
||||
"nav.chat": "Chat",
|
||||
"nav.brain": "Cervello",
|
||||
"nav.profiling": "Profiling",
|
||||
|
||||
"brand.tagline": "gigante locale, impronta minima",
|
||||
|
||||
"sidebar.connection": "Connessione",
|
||||
"sidebar.endpoint": "Endpoint API",
|
||||
"sidebar.apiKey": "Chiave API",
|
||||
"sidebar.apiKeyPlaceholder": "opzionale",
|
||||
"sidebar.apiKeyHelp": "Conservata solo in memoria · inviata a questo endpoint",
|
||||
"sidebar.probe": "Sonda il server",
|
||||
"status.connected": "Motore raggiungibile",
|
||||
"status.notConnected": "Non connesso",
|
||||
"status.runtimeUnavailable": "Metriche runtime non disponibili",
|
||||
"status.serverError": "Impossibile raggiungere il server.",
|
||||
"status.generationFailed": "Generazione fallita.",
|
||||
|
||||
"sidebar.runtime": "Runtime",
|
||||
"sidebar.runtimeProbe": "Sonda il server per ispezionare lo stato runtime.",
|
||||
"sidebar.schedulerOnline": "Scheduler online",
|
||||
"dashboard.active": "Attive",
|
||||
"dashboard.queued": "In coda",
|
||||
"dashboard.completed": "Completate",
|
||||
"dashboard.failures": "Fallite",
|
||||
"dashboard.session": "Sessione:",
|
||||
"dashboard.prompt": "prompt",
|
||||
"dashboard.completion": "completion",
|
||||
|
||||
"tier.vram": "VRAM",
|
||||
"tier.ram": "RAM",
|
||||
"tier.disk": "Disco",
|
||||
"tier.ariaLabel": "Expert: {{vram}} VRAM, {{ram}} RAM, {{disk}} disco",
|
||||
|
||||
"sidebar.inference": "Inferenza",
|
||||
"sidebar.model": "Modello",
|
||||
"sidebar.kvSession": "Sessione KV",
|
||||
"sidebar.kvSessionHelp": "Contesto isolato · la conversazione segue lo slot selezionato",
|
||||
"sidebar.sessionLabel": "Sessione {{slot}}",
|
||||
"sidebar.temperature": "Temperatura",
|
||||
"sidebar.maxTokens": "Token di output massimi",
|
||||
"sidebar.reasoning": "Ragionamento",
|
||||
"sidebar.transport": "Trasporto compatibile OpenAI",
|
||||
|
||||
"topbar.activeModel": "MODELLO ATTIVO",
|
||||
"topbar.tokens": "{{n}} token",
|
||||
"topbar.tokPerSec": "{{n}} tok/s",
|
||||
"topbar.slot": "slot {{n}}",
|
||||
"topbar.clear": "Pulisci",
|
||||
|
||||
"hero.title": "MOTORE COLIBRÌ",
|
||||
"hero.subtitle": "Interroga il gigante.",
|
||||
"hero.tagline": "La macchina resta tua.",
|
||||
"hero.description": "Connettiti a un server colibrì locale e ricevi le risposte in streaming direttamente dal tuo hardware. Nulla lascia l'endpoint che scegli.",
|
||||
"prompts.routing": "Spiega come funziona il routing degli expert",
|
||||
"prompts.benchmark": "Scrivi un piccolo benchmark in C",
|
||||
"prompts.caching": "Confronta il caching RAM e VRAM",
|
||||
|
||||
"chat.you": "Tu",
|
||||
"chat.colibri": "colibrì",
|
||||
"chat.placeholder": "Scrivi a colibrì…",
|
||||
"chat.inputHint": "Invio per inviare · Shift+Invio per andare a capo",
|
||||
"chat.stop": "Ferma la generazione",
|
||||
"chat.send": "Invia messaggio",
|
||||
|
||||
"brain.title": "Corteccia degli expert",
|
||||
"brain.waiting": "in attesa del motore",
|
||||
"brain.layers": "{{rows}} layer × {{cols}} expert",
|
||||
"brain.brightnessHint": "luminosità = calore di routing",
|
||||
"brain.flashHint": "⚡ flash bianco = instradato in questo turno",
|
||||
"brain.connectHint": "Connettiti al motore per vedere la corteccia.",
|
||||
"brain.neverRouted": "mai instradato",
|
||||
"brain.selections": "~2^{{heat}} selezioni",
|
||||
"brain.specialist": "⭐ Specialista: {{top}}",
|
||||
"brain.generalist": "Generalista",
|
||||
"brain.mtp": "Testa MTP — prepara il prossimo token per la decodifica speculativa",
|
||||
"brain.early": "layer iniziali — caratteristiche superficiali: token, ortografia, sintassi locale",
|
||||
"brain.lowerMiddle": "layer medio-bassi — struttura frasale, relazioni tra parole, fatti semplici",
|
||||
"brain.upperMiddle": "layer medio-alti — semantica, contesto a lungo raggio, passi di ragionamento",
|
||||
"brain.late": "layer avanzati — pianificazione della risposta, stile, coerenza",
|
||||
"brain.final": "layer finali — formazione dell'output: scelta della distribuzione next-token",
|
||||
|
||||
"profile.title": "Profiling — dove il motore spende ogni turno",
|
||||
"profile.ioWait": "Attesa I/O",
|
||||
"profile.expertMatmul": "Matmul expert",
|
||||
"profile.attention": "Attenzione",
|
||||
"profile.lmHead": "LM head",
|
||||
"profile.other": "Altro",
|
||||
"profile.empty": "Nessun turno profilato — invia un messaggio e i dettagli appariranno qui.",
|
||||
"profile.connectHint": "Connettiti al motore per raccogliere i tempi per turno.",
|
||||
"profile.lastTurn": "Ultimo turno",
|
||||
"profile.wallTime": "Tempo totale",
|
||||
"profile.batching": "Batching",
|
||||
"profile.tokensPerForward": "token / forward",
|
||||
"profile.diskService": "Servizio disco",
|
||||
"profile.overlapped": "sovrapposto al calcolo",
|
||||
"profile.window": "Finestra · ultimi {{n}} turni",
|
||||
"profile.throughputTitle": "Throughput per turno (tok/s)",
|
||||
"profile.phaseTitle": "Tempo per turno per fase (s)",
|
||||
"profile.turnCol": "Turno",
|
||||
"profile.tokensCol": "Token",
|
||||
"profile.wallCol": "Totale",
|
||||
"profile.turnsLabel": "{{n}} turni · dal meno al più recente",
|
||||
"profile.oneTurn": "1 turno",
|
||||
"profile.diskNote": "Il servizio disco è il tempo speso a leggere gli expert sui thread I/O; si sovrappone al calcolo, quindi solo l'attesa I/O effettivamente percepita dal thread di calcolo conta nella ripartizione del tempo totale. Con più sessioni KV, le quote descrivono l'intero motore nella finestra del turno.",
|
||||
|
||||
"error.title": "L'interfaccia colibrì ha riscontrato un errore",
|
||||
"error.hint": "Il motore non è stato coinvolto. Prova a ricaricare la pagina.",
|
||||
"error.retry": "Riprova",
|
||||
}
|
||||
|
||||
export default it
|
||||
@@ -0,0 +1,113 @@
|
||||
const zhCN: Record<string, string> = {
|
||||
"nav.chat": "对话",
|
||||
"nav.brain": "大脑",
|
||||
"nav.profiling": "性能分析",
|
||||
|
||||
"brand.tagline": "本地巨人，极小足迹",
|
||||
|
||||
"sidebar.connection": "连接",
|
||||
"sidebar.endpoint": "API 端点",
|
||||
"sidebar.apiKey": "API 密钥",
|
||||
"sidebar.apiKeyPlaceholder": "可选",
|
||||
"sidebar.apiKeyHelp": "仅保存在内存中 · 发送到此端点",
|
||||
"sidebar.probe": "探测服务器",
|
||||
"status.connected": "引擎已连接",
|
||||
"status.notConnected": "未连接",
|
||||
"status.runtimeUnavailable": "运行时指标不可用",
|
||||
"status.serverError": "无法连接到服务器。",
|
||||
"status.generationFailed": "生成失败。",
|
||||
|
||||
"sidebar.runtime": "运行时",
|
||||
"sidebar.runtimeProbe": "探测服务器以查看运行时状态。",
|
||||
"sidebar.schedulerOnline": "调度器在线",
|
||||
"dashboard.active": "活跃",
|
||||
"dashboard.queued": "排队",
|
||||
"dashboard.completed": "已完成",
|
||||
"dashboard.failures": "失败",
|
||||
"dashboard.session": "会话：",
|
||||
"dashboard.prompt": "提示词",
|
||||
"dashboard.completion": "补全",
|
||||
|
||||
"tier.vram": "VRAM",
|
||||
"tier.ram": "RAM",
|
||||
"tier.disk": "磁盘",
|
||||
"tier.ariaLabel": "专家分布：{{vram}} VRAM、{{ram}} RAM、{{disk}} 磁盘",
|
||||
|
||||
"sidebar.inference": "推理",
|
||||
"sidebar.model": "模型",
|
||||
"sidebar.kvSession": "KV 会话",
|
||||
"sidebar.kvSessionHelp": "独立上下文 · 对话跟随所选槽位",
|
||||
"sidebar.sessionLabel": "会话 {{slot}}",
|
||||
"sidebar.temperature": "温度",
|
||||
"sidebar.maxTokens": "最大输出 token 数",
|
||||
"sidebar.reasoning": "推理模式",
|
||||
"sidebar.transport": "OpenAI 兼容协议",
|
||||
|
||||
"topbar.activeModel": "当前模型",
|
||||
"topbar.tokens": "{{n}} tokens",
|
||||
"topbar.tokPerSec": "{{n}} tok/s",
|
||||
"topbar.slot": "槽位 {{n}}",
|
||||
"topbar.clear": "清空",
|
||||
|
||||
"hero.title": "COLIBRÌ 引擎",
|
||||
"hero.subtitle": "向巨人提问。",
|
||||
"hero.tagline": "让机器属于你。",
|
||||
"hero.description": "连接到本地 colibrì 服务器，直接从你的硬件流式获取响应。所有数据都留在你选择的端点内。",
|
||||
"prompts.routing": "解释专家路由是如何工作的",
|
||||
"prompts.benchmark": "写一个简单的 C 基准测试",
|
||||
"prompts.caching": "比较 RAM 和 VRAM 缓存",
|
||||
|
||||
"chat.you": "你",
|
||||
"chat.colibri": "colibrì",
|
||||
"chat.placeholder": "给 colibrì 发消息…",
|
||||
"chat.inputHint": "回车发送 · Shift+回车换行",
|
||||
"chat.stop": "停止生成",
|
||||
"chat.send": "发送消息",
|
||||
|
||||
"brain.title": "专家皮层",
|
||||
"brain.waiting": "等待引擎连接",
|
||||
"brain.layers": "{{rows}} 层 × {{cols}} 专家",
|
||||
"brain.brightnessHint": "亮度 = 路由热度",
|
||||
"brain.flashHint": "⚡ 白色闪烁 = 本轮被路由",
|
||||
"brain.connectHint": "连接引擎以查看皮层。",
|
||||
"brain.neverRouted": "从未被路由",
|
||||
"brain.selections": "约 2^{{heat}} 次选择",
|
||||
"brain.specialist": "⭐ 专精：{{top}}",
|
||||
"brain.generalist": "通用型",
|
||||
"brain.mtp": "MTP 头 — 为投机解码起草下一个 token",
|
||||
"brain.early": "早期层 — 表面特征：token、拼写、局部语法",
|
||||
"brain.lowerMiddle": "中低层 — 短语结构、词语关系、简单事实",
|
||||
"brain.upperMiddle": "中高层 — 语义、长距离上下文、推理步骤",
|
||||
"brain.late": "后期层 — 规划答案、风格、连贯性",
|
||||
"brain.final": "末尾层 — 输出成型：选择实际的 next-token 分布",
|
||||
|
||||
"profile.title": "性能分析 — 引擎每轮的时间花在哪里",
|
||||
"profile.ioWait": "I/O 等待",
|
||||
"profile.expertMatmul": "专家矩阵乘",
|
||||
"profile.attention": "注意力",
|
||||
"profile.lmHead": "LM head",
|
||||
"profile.other": "其他",
|
||||
"profile.empty": "暂无性能数据 — 发送一条消息，分析结果将显示在这里。",
|
||||
"profile.connectHint": "连接引擎以采集每轮耗时。",
|
||||
"profile.lastTurn": "最近一轮",
|
||||
"profile.wallTime": "总耗时",
|
||||
"profile.batching": "批处理",
|
||||
"profile.tokensPerForward": "tokens / 前向",
|
||||
"profile.diskService": "磁盘服务",
|
||||
"profile.overlapped": "与计算重叠",
|
||||
"profile.window": "窗口 · 最近 {{n}} 轮",
|
||||
"profile.throughputTitle": "每轮吞吐量 (tok/s)",
|
||||
"profile.phaseTitle": "每轮各阶段耗时 (s)",
|
||||
"profile.turnCol": "轮次",
|
||||
"profile.tokensCol": "Tokens",
|
||||
"profile.wallCol": "总耗时",
|
||||
"profile.turnsLabel": "{{n}} 轮 · 从旧到新",
|
||||
"profile.oneTurn": "1 轮",
|
||||
"profile.diskNote": "磁盘服务是在 I/O 线程上读取专家的时间；它与计算重叠，因此只有计算线程实际感受到的 I/O 等待 才计入总耗时分解。多 KV 会话时，份额描述的是整个引擎在该轮窗口内的表现。",
|
||||
|
||||
"error.title": "colibrì UI 遇到错误",
|
||||
"error.hint": "引擎不受影响。请尝试刷新页面。",
|
||||
"error.retry": "重试",
|
||||
}
|
||||
|
||||
export default zhCN
|
||||
@@ -0,0 +1,113 @@
|
||||
const zhTW: Record<string, string> = {
|
||||
"nav.chat": "對話",
|
||||
"nav.brain": "大腦",
|
||||
"nav.profiling": "效能分析",
|
||||
|
||||
"brand.tagline": "本地巨人，極小足跡",
|
||||
|
||||
"sidebar.connection": "連線",
|
||||
"sidebar.endpoint": "API 端點",
|
||||
"sidebar.apiKey": "API 金鑰",
|
||||
"sidebar.apiKeyPlaceholder": "選填",
|
||||
"sidebar.apiKeyHelp": "僅保存在記憶體中 · 傳送到此端點",
|
||||
"sidebar.probe": "探測伺服器",
|
||||
"status.connected": "引擎已連線",
|
||||
"status.notConnected": "未連線",
|
||||
"status.runtimeUnavailable": "執行階段指標不可用",
|
||||
"status.serverError": "無法連線到伺服器。",
|
||||
"status.generationFailed": "生成失敗。",
|
||||
|
||||
"sidebar.runtime": "執行階段",
|
||||
"sidebar.runtimeProbe": "探測伺服器以檢視執行階段狀態。",
|
||||
"sidebar.schedulerOnline": "排程器上線",
|
||||
"dashboard.active": "進行中",
|
||||
"dashboard.queued": "排隊中",
|
||||
"dashboard.completed": "已完成",
|
||||
"dashboard.failures": "失敗",
|
||||
"dashboard.session": "工作階段：",
|
||||
"dashboard.prompt": "提示詞",
|
||||
"dashboard.completion": "補全",
|
||||
|
||||
"tier.vram": "VRAM",
|
||||
"tier.ram": "RAM",
|
||||
"tier.disk": "磁碟",
|
||||
"tier.ariaLabel": "專家分佈：{{vram}} VRAM、{{ram}} RAM、{{disk}} 磁碟",
|
||||
|
||||
"sidebar.inference": "推論",
|
||||
"sidebar.model": "模型",
|
||||
"sidebar.kvSession": "KV 工作階段",
|
||||
"sidebar.kvSessionHelp": "獨立上下文 · 對話跟隨所選插槽",
|
||||
"sidebar.sessionLabel": "工作階段 {{slot}}",
|
||||
"sidebar.temperature": "溫度",
|
||||
"sidebar.maxTokens": "最大輸出 token 數",
|
||||
"sidebar.reasoning": "推理模式",
|
||||
"sidebar.transport": "OpenAI 相容協定",
|
||||
|
||||
"topbar.activeModel": "目前模型",
|
||||
"topbar.tokens": "{{n}} tokens",
|
||||
"topbar.tokPerSec": "{{n}} tok/s",
|
||||
"topbar.slot": "插槽 {{n}}",
|
||||
"topbar.clear": "清除",
|
||||
|
||||
"hero.title": "COLIBRÌ 引擎",
|
||||
"hero.subtitle": "向巨人提問。",
|
||||
"hero.tagline": "讓機器屬於你。",
|
||||
"hero.description": "連線到本地 colibrì 伺服器，直接從你的硬體串流取得回應。所有資料都留在你選擇的端點內。",
|
||||
"prompts.routing": "解釋專家路由如何運作",
|
||||
"prompts.benchmark": "撰寫一個簡單的 C 基準測試",
|
||||
"prompts.caching": "比較 RAM 與 VRAM 快取",
|
||||
|
||||
"chat.you": "你",
|
||||
"chat.colibri": "colibrì",
|
||||
"chat.placeholder": "傳送訊息給 colibrì…",
|
||||
"chat.inputHint": "Enter 傳送 · Shift+Enter 換行",
|
||||
"chat.stop": "停止生成",
|
||||
"chat.send": "傳送訊息",
|
||||
|
||||
"brain.title": "專家皮層",
|
||||
"brain.waiting": "等待引擎連線",
|
||||
"brain.layers": "{{rows}} 層 × {{cols}} 專家",
|
||||
"brain.brightnessHint": "亮度 = 路由熱度",
|
||||
"brain.flashHint": "⚡ 白色閃爍 = 本輪被路由",
|
||||
"brain.connectHint": "連線引擎以檢視皮層。",
|
||||
"brain.neverRouted": "從未被路由",
|
||||
"brain.selections": "約 2^{{heat}} 次選擇",
|
||||
"brain.specialist": "⭐ 專精：{{top}}",
|
||||
"brain.generalist": "通用型",
|
||||
"brain.mtp": "MTP 頭 — 為推測式解碼起草下一個 token",
|
||||
"brain.early": "早期層 — 表面特徵：token、拼寫、局部語法",
|
||||
"brain.lowerMiddle": "中低層 — 片語結構、詞語關係、簡單事實",
|
||||
"brain.upperMiddle": "中高層 — 語意、長距離上下文、推理步驟",
|
||||
"brain.late": "後期層 — 規劃答案、風格、連貫性",
|
||||
"brain.final": "末尾層 — 輸出成型：選擇實際的 next-token 分佈",
|
||||
|
||||
"profile.title": "效能分析 — 引擎每輪的時間花在哪裡",
|
||||
"profile.ioWait": "I/O 等待",
|
||||
"profile.expertMatmul": "專家矩陣乘",
|
||||
"profile.attention": "注意力",
|
||||
"profile.lmHead": "LM head",
|
||||
"profile.other": "其他",
|
||||
"profile.empty": "尚無效能數據 — 傳送一則訊息，分析結果將顯示在這裡。",
|
||||
"profile.connectHint": "連線引擎以採集每輪耗時。",
|
||||
"profile.lastTurn": "最近一輪",
|
||||
"profile.wallTime": "總耗時",
|
||||
"profile.batching": "批次處理",
|
||||
"profile.tokensPerForward": "tokens / 前向",
|
||||
"profile.diskService": "磁碟服務",
|
||||
"profile.overlapped": "與運算重疊",
|
||||
"profile.window": "視窗 · 最近 {{n}} 輪",
|
||||
"profile.throughputTitle": "每輪吞吐量 (tok/s)",
|
||||
"profile.phaseTitle": "每輪各階段耗時 (s)",
|
||||
"profile.turnCol": "輪次",
|
||||
"profile.tokensCol": "Tokens",
|
||||
"profile.wallCol": "總耗時",
|
||||
"profile.turnsLabel": "{{n}} 輪 · 從舊到新",
|
||||
"profile.oneTurn": "1 輪",
|
||||
"profile.diskNote": "磁碟服務是在 I/O 執行緒上讀取專家的時間；它與運算重疊，因此只有運算執行緒實際感受到的 I/O 等待 才計入總耗時分解。多 KV 工作階段時，份額描述的是整個引擎在該輪視窗內的表現。",
|
||||
|
||||
"error.title": "colibrì UI 遇到錯誤",
|
||||
"error.hint": "引擎不受影響。請嘗試重新整理頁面。",
|
||||
"error.retry": "重試",
|
||||
}
|
||||
|
||||
export default zhTW
|
||||
+3
-1
@@ -64,7 +64,9 @@ button:focus-visible, input:focus-visible, textarea:focus-visible, select:focus-
|
||||
.toggle-row { display: flex; align-items: center; justify-content: space-between; height: 42px; padding: 0 11px; border: 1px solid var(--border); border-radius: 9px; color: #a9b4b8; background: var(--input); }
|
||||
.toggle-row > span { display: flex; align-items: center; gap: 8px; font-size: 11px; font-weight: 600; }.toggle-row.active { border-color: rgba(78,214,165,.35); color: var(--foreground); }
|
||||
.toggle-row i { width: 30px; height: 17px; padding: 2px; border-radius: 20px; background: #293136; transition: .2s; }.toggle-row i b { display: block; width: 13px; height: 13px; border-radius: 50%; background: #78858a; transition: .2s; }.toggle-row.active i { background: rgba(78,214,165,.28); }.toggle-row.active i b { transform: translateX(13px); background: var(--primary); }
|
||||
.sidebar-foot { margin-top: auto; display: flex; align-items: center; gap: 7px; color: #59666b; font-size: 10px; }
|
||||
.sidebar-foot { margin-top: auto; display: flex; flex-direction: column; gap: 6px; color: #59666b; font-size: 10px; }
|
||||
.sidebar-foot > div { display: flex; align-items: center; gap: 7px; }
|
||||
.locale-switcher select { background: transparent; border: 1px solid var(--border); border-radius: 4px; color: inherit; font-size: 10px; padding: 2px 4px; cursor: pointer; }
|
||||
|
||||
.chat-panel { min-width: 0; height: 100vh; display: grid; grid-template-rows: 72px minmax(0, 1fr) auto; }
|
||||
.topbar { display: flex; align-items: center; justify-content: space-between; padding: 0 32px; border-bottom: 1px solid var(--border); }
|
||||
|
||||
+4
-1
@@ -2,10 +2,13 @@ import { createRoot } from "react-dom/client"
|
||||
|
||||
import App from "./App"
|
||||
import { ErrorBoundary } from "./ErrorBoundary"
|
||||
import { LocaleProvider } from "./i18n"
|
||||
import "./index.css"
|
||||
|
||||
createRoot(document.getElementById("root")!).render(
|
||||
<ErrorBoundary>
|
||||
<App />
|
||||
<LocaleProvider>
|
||||
<App />
|
||||
</LocaleProvider>
|
||||
</ErrorBoundary>,
|
||||
)
|
||||
|
||||
Reference in New Issue
Block a user