diff --git a/.gitignore b/.gitignore index 203a322..981e740 100644 --- a/.gitignore +++ b/.gitignore @@ -31,6 +31,10 @@ c/tests/test_tier c/tests/test_tier.exe c/tests/test_grammar c/tests/test_grammar.exe +c/tests/test_schema_gbnf +c/tests/test_schema_gbnf.exe +c/tests/test_compat_direct +c/tests/test_compat_direct.exe # oracoli tiny generati (make_glm_oracle.py) e dati benchmark scaricati c/glm_tiny/ @@ -60,3 +64,7 @@ c/backend_metal_test c/tests/test_tier c/mio_env/ c/bench/ +c/tests/test_decode_batch +c/tests/test_i4_acc512 +c/tests/test_idot +c/tests/test_uring diff --git a/README.md b/README.md index f72c4e9..2e22048 100644 --- a/README.md +++ b/README.md @@ -17,14 +17,40 @@ $ ./coli chat ◆ Ciao! 😊 Come posso aiutarti oggi? ``` + +## See it running + +

+ colibrì web dashboard — live metrics, hardware panel, expert tiers +

+

The web dashboard (./coli web): a 744B model answering at 4+ tok/s end-to-end on 6× RTX 5090 — +with live token metrics, the hardware panel, and the VRAM/RAM/disk expert tiers.

+ +

+ the Brain page — 19,456 experts as a live cortex +

+

The Brain page: all 19,456 experts as a living cortex — colour is the storage tier, +brightness is routing heat, and every expert routed in a turn flashes white. Hovering shows the expert's +measured topic affinity.

+ +## Contents + +- [The idea](#the-idea) +- [See it running](#see-it-running) +- [What's implemented](#whats-implemented) +- [Honest numbers](#honest-numbers-wsl2-12-cores-25-gb-ram-nvme-via-vhdx) +- [Download the model](#download-the-model) +- [Web dashboard](#web-dashboard) +- [Got a better machine?](#got-a-better-machine-try-it--heres-what-to-expect) + ## The idea A 744B Mixture-of-Experts model activates only ~40B parameters per token — and only ~11 GB of those change from token to token (the routed experts). So: - the **dense part** (attention, shared experts, embeddings — ~17B params) stays **resident in RAM at int4** (~9.9 GB); -- the **21,504 routed experts** (75 MoE layers × 256 experts + the MTP head, ~19 MB each at int4) live **on disk** (~370 GB) and are **streamed on demand**, with a per-layer LRU cache, an optional pinned hot-store, and the OS page cache as a free L2. +- the **19,456 routed experts** (75 MoE layers × 256 experts + the MTP head, ~19 MB each at int4) live **on disk** (~370 GB) and are **streamed on demand**, with a per-layer LRU cache, an optional pinned hot-store, and the OS page cache as a free L2. -The engine is a single C file (`c/glm.c`, ~2,400 lines) plus small headers. No BLAS, no Python at runtime, no GPU required (an opt-in CUDA tier for pinned experts exists — see below). +The engine is a single C file (`c/glm.c`) plus small headers. No BLAS, no Python at runtime, no GPU required (an opt-in CUDA tier for pinned experts exists — see below). ## What's implemented @@ -354,6 +380,10 @@ PIN=stats.txt PIN_GB=160 SNAP=/nvme/glm52_i4 ./glm 64 4 4 COLI_CUDA=1 COLI_GPUS=0,1,2,3,4,5 CUDA_EXPERT_GB=150 \ CUDA_DENSE=1 PIN=stats.txt PIN_GB=300 RAM_GB=226 \ SNAP=/nvme/glm52_i4 ./glm 64 4 4 +# large-RAM host: fill safe VRAM, then keep every remaining expert in RAM +COLI_CUDA=1 COLI_GPUS=0,1,2,3,4,5 CUDA_EXPERT_GB=auto \ +CUDA_DENSE=1 COLI_CUDA_ATTN=1 PIN=stats.txt PIN_GB=all RAM_GB=auto \ +SNAP=/nvme/glm52_i4 ./glm 64 4 4 ``` Selected experts are uploaded during startup, so capacity failures occur before @@ -375,6 +405,25 @@ replay from 1.87 to 2.16 tok/s (+15.7%), reduced expert disk wait from 5.144s to 3.948s, and kept the projected RAM peak below `RAM_GB=226`. The cache cap adjusts down automatically (54 to 40 in that run) so the larger pinned tier does not exceed the process budget. Start lower on hosts with less available RAM. + +`CUDA_EXPERT_GB=auto` fills each selected device only up to its measured free +memory minus projected dense tensors and 2 GB of runtime headroom. `PIN_GB=all` +then loads every remaining routed expert into RAM, eliminating decode-time disk +misses when the host budget permits it. The regular `RAM_GB` guard still clamps +the per-layer working cache and rejects unsafe projections; this mode is intended +for dedicated high-memory inference hosts, not desktops running other workloads. +On a dedicated 251 GiB host with six RTX 5090s, this mode selected a 176.7 GB +VRAM expert tier and a 191.3 GB RAM tier (all 19,456 experts resident). The +mode also adapts the VRAM tier every 16 emitted tokens by swapping hot RAM +experts into existing GPU slots. A real 64-token greedy GLM-5.2 generation +measured **6.00 tok/s decode**, up from +2.20 tok/s end-to-end with the earlier 150 GB tier; expert hit rate was 100% +and disk wait was zero. Prompt prefill is reported separately. This is a +host-specific capacity result, not a portable default. + +Text-mode timing reports prefill separately from decode. The decode rate starts +after the prompt KV is built, so it is comparable to `REPLAY` throughput without +hiding time-to-first-token. MTP speculation defaults off on CUDA because cold draft routes increase expert traffic; an explicit `DRAFT=n` still overrides the default. @@ -406,8 +455,8 @@ CUDA, CPU hot-store, and CUDA hot-expert execution with identical replay tokens. ### Web interface -`web/` contains a community-contributed browser UI (React + TypeScript, ~390 -lines of source, a pure API client — it never touches the engine directly): +`web/` contains a community-contributed browser UI (React + TypeScript, a pure +API client — it never touches the engine directly): ```bash cd web @@ -419,7 +468,7 @@ works against the colibrì OpenAI-compatible server (in review, #21) or any othe compatible endpoint. Nothing leaves the endpoint you configure. The terminal `coli chat` remains the first-class interface. -Useful knobs (env or flags): `--temp T` token sampling temperature (default 0.7 + nucleus 0.90 — tuned for int4; 0 = greedy), `--topp 0.7` adaptive expert top-p (30–40% less disk), `--ngen N` max tokens per answer (`:more` in chat continues a truncated one), `--repin N` adapt RAM/VRAM hot experts every N emitted tokens, `AUTOPIN=0` disable the learning cache's auto-pin, `THINK=1` enable GLM-5.2's reasoning block, `DRAFT=n` MTP draft depth, `GRAMMAR=g.gbnf` grammar-forced drafts for constrained JSON/NDJSON output (`GRAMMAR_DRAFT=n` caps the forced span), `TF=1` teacher-forcing validation, `PILOT=1` router-lookahead disk prefetch (experimental — see below), `CAP_RAISE=0` don't auto-grow the expert cache. +Useful knobs (env or flags): `--temp T` token sampling temperature (default 0.7 + nucleus 0.90 — tuned for int4; 0 = greedy), `--topp 0.7` adaptive expert top-p (30–40% less disk), `--ngen N` max tokens per answer (`:more` in chat continues a truncated one), `--repin N` adapt RAM/VRAM hot experts every N emitted tokens, `AUTOPIN=0` disable the learning cache's auto-pin, `THINK=1` enable GLM-5.2's reasoning block, `DRAFT=n` MTP draft depth, `GRAMMAR=g.gbnf` grammar-forced drafts for constrained JSON/NDJSON output (`GRAMMAR_DRAFT=n` caps the forced span), `TF=1` teacher-forcing validation, `PILOT=1` router-lookahead disk prefetch (experimental — see below), `URING=1` Linux-only batched expert I/O (implies `PIPE=1`; also batches `PILOT_REAL`), `PIPE=0` disable the async expert-load pool (**default ON on Windows** — overlaps expert `pread` with the matmul so the CPU isn't idle waiting on the SSD; measured −18% disk service time), `RAM_GB=` claim more RAM for the expert cache than the conservative auto-detect (e.g. `RAM_GB=31` on a 32 GB host raises the cache cap and hit rate measurably), `CAP_RAISE=0` don't auto-grow the expert cache. ### Resource policy @@ -444,10 +493,10 @@ Disk is an immutable recovery source, not a normal decode target. If the plan leaves cold expert bytes on disk, speed depends on cache hit rate; output quality does not. -Cold expert reads use a deferred pipeline: resident RAM/VRAM experts execute +Cold expert reads can use a deferred pipeline: resident RAM/VRAM experts execute while missing experts are loaded in a bounded background I/O pool, then the -cold results join before the layer completes. `IO_THREADS=n` overrides the -default eight loader threads when foreground work exists. Profiling reports +cold results join before the layer completes. The pool engages only under +`PIPE=1`; `PIPE_WORKERS=n` sets its worker count (default 8). Profiling reports both disk service time and the smaller foreground-visible wait time so overlap is explicit rather than credited as unexplained speedup. @@ -478,6 +527,23 @@ thrashing. Persistent `.coli_usage` remains the long-term signal and is not deca **Conversations reopen warm** (`.coli_kv`, since 2026-07-10): `coli chat` persists the compressed MLA KV-cache to disk after every turn (~182 KB/token, appended incrementally, crash-safe). Close the chat, reopen it tomorrow — the model still remembers the whole conversation and **zero re-prefill happens**: validated byte-identical to an uninterrupted session. `:reset` clears it, `KVSAVE=0` disables it. +## Web dashboard + +One command serves the OpenAI-compatible API **and** the web console on the same port, then opens your browser when the engine is ready: + +```bash +cd web && npm install && npm run build # once +./coli web --model +``` + +What you get: + +- **Chat** with live metrics: a flashing token counter while generating, then tok/s, time-to-first-token, prompt→completion counts and queue wait; +- **Runtime panel**: your hardware (CPU, GPUs + VRAM, RAM, cores), the scheduler, and the live expert-tier bar — how many of the 19,456 experts sit in VRAM / RAM / disk right now; +- **Brain**: the whole model as a 76×256 cortex, one cell per expert. Colour = tier, brightness = routing heat, and the experts routed in each turn flash white and decay — you watch the model think. Hover any cell for its tier, heat and [measured topic affinity](https://github.com/JustVugg/colibri/issues/175) (specialists for code, Chinese, math, law… live in layers 11–22). + +The dashboard talks to the engine over two tiny protocol lines (`TIERS`, `EMAP`/`HITS`) and plain JSON endpoints — nothing heavier than the engine itself. + ## Got a better machine? Try it — here's what to expect colibrì was built on deliberately humble hardware (12 cores, 25 GB RAM, an older DRAM-less NVMe behind a WSL2 VHDX that measured ~1 GB/s random on *this* drive — note WSL2 VHDX is not inherently slow: a community 5090 box measured 10.5 GB/s O_DIRECT through one, [#101](https://github.com/JustVugg/colibri/issues/101)). **Every one of those constraints is a knob your machine can turn up.** The engine needs: Linux (or WSL2), macOS, or **Windows 11 natively (MinGW-w64)**; gcc with OpenMP, AVX2, ≥16 GB RAM, and the ~370 GB int4 model on a local NVMe (ext4/NTFS — never a network/9p mount). @@ -591,6 +657,7 @@ c/ ├── scripts/ long-running conversion helpers └── tests/ dependency-free C and Python tests web/ browser UI (pure OpenAI-API client, community-maintained) +desktop/ Tauri v2 desktop shell wrapping the web UI ``` The runtime path intentionally stays flat and readable: `glm.c` plus its small diff --git a/c/Makefile b/c/Makefile index c1e9319..5ae7a9c 100644 --- a/c/Makefile +++ b/c/Makefile @@ -14,7 +14,7 @@ TRIPLET := $(shell $(DETECT_CC) -dumpmachine 2>/dev/null) MINGW := $(findstring mingw,$(TRIPLET)) CYGWIN := $(findstring cygwin,$(TRIPLET)) DARWIN := $(findstring darwin,$(TRIPLET)) -PPC64 := $(findstring powerpc64,$(TRIPLET)) +LINUX := $(findstring linux,$(TRIPLET)) IS_WIN := $(MINGW)$(CYGWIN) # Fallbacks for the rare toolchain that does not answer -dumpmachine: keep the @@ -26,10 +26,17 @@ ifeq ($(IS_WIN),) UNAME_S := $(shell uname -s) UNAME_M := $(shell uname -m) DARWIN := $(findstring Darwin,$(UNAME_S)) -PPC64 := $(findstring ppc64,$(UNAME_M)) endif endif +TARGET_CPU := $(firstword $(subst -, ,$(TRIPLET))) +ifeq ($(TARGET_CPU),) +TARGET_CPU := $(UNAME_M) +endif +X86_64 := $(filter x86_64 amd64,$(TARGET_CPU)) +AARCH64 := $(filter aarch64 arm64,$(TARGET_CPU)) +PPC64 := $(filter powerpc64% ppc64%,$(TARGET_CPU)) + ifneq (,$(DARWIN)) # --- macOS / Apple Silicon --- # Apple clang non include il runtime OpenMP: se c'e' libomp di Homebrew lo usa @@ -37,7 +44,9 @@ ifneq (,$(DARWIN)) # Niente -march: su arm64 NEON e' baseline (i kernel __ARM_NEON si attivano da soli). CC = clang OMPDIR := $(shell brew --prefix libomp 2>/dev/null) -ifneq ($(OMPDIR),) +# `brew --prefix libomp` can print the formula's prospective path even when it +# is not installed, so verify both artifacts before adding unusable flags. +ifneq ($(and $(OMPDIR),$(wildcard $(OMPDIR)/include/omp.h),$(wildcard $(OMPDIR)/lib/libomp.*)),) OMPC = -Xclang -fopenmp -I$(OMPDIR)/include OMPL = -L$(OMPDIR)/lib -lomp else @@ -46,6 +55,12 @@ OMPC = OMPL = endif CFLAGS = -O3 $(OMPC) -Wall -Wextra -Wno-unused-parameter -Wno-misleading-indentation -Wno-unused-function +# Opt-in: ARCH=native appends -mcpu=native (arm64 clang uses -mcpu, not -march), +# which unlocks the i8mm SMMLA int8/int4 dot kernels in glm.c. ARCH unset -> +# no -mcpu, default build byte-identical. Apple clang knows apple-m4 / native. +ifneq ($(ARCH),) +CFLAGS += -mcpu=$(ARCH) +endif LDFLAGS = -lm $(OMPL) EXE = else ifneq ($(IS_WIN),) @@ -60,7 +75,11 @@ else ifneq ($(IS_WIN),) CC = gcc ARCH ?= x86-64-v3 CFLAGS = -D_FILE_OFFSET_BITS=64 -O3 -march=$(ARCH) -fopenmp -Wall -Wextra -Wno-unused-parameter -Wno-misleading-indentation -Wno-unused-function -LDFLAGS = -lm -fopenmp -static +# -lpsapi: compat.h calls GetProcessMemoryInfo (rss_gb). It's linked via +# #pragma comment(lib,"psapi.lib") for MSVC, but MinGW gcc ignores that pragma +# (warns), so link psapi explicitly or the build fails undefined-reference on +# modern GCC (e.g. 16.x under UCRT). Harmless where the pragma also resolves it. +LDFLAGS = -lm -fopenmp -static -lpsapi EXE = .exe else ifneq (,$(PPC64)) @@ -70,18 +89,19 @@ ifneq (,$(PPC64)) # (validated token-exact vs the transformers oracle on a POWER8 S824). CC = gcc ARCH ?= native -CFLAGS = -O3 -mcpu=$(ARCH) -fopenmp -Wall -Wextra -Wno-unused-parameter -Wno-misleading-indentation -Wno-unused-function -LDFLAGS = -lm -fopenmp +CFLAGS = -O3 -mcpu=$(ARCH) -fopenmp -pthread -Wall -Wextra -Wno-unused-parameter -Wno-misleading-indentation -Wno-unused-function +LDFLAGS = -lm -fopenmp -pthread EXE = else -# --- Linux x86-64 (percorso originale, invariato) --- +# --- Linux / *BSD x86-64 (percorso originale, invariato) --- CC = gcc # ARCH=native -> ottimizzato per QUESTA macchina (default, piu' veloce). # ARCH=x86-64-v3 -> binario PORTABILE su qualsiasi x86-64 moderno con AVX2 (per distribuire). # ARCH=x86-64 -> massima compatibilita' (niente AVX2: usa il path scalare di fallback). +# -pthread: Linux lo tira dentro via -fopenmp, i *BSD no e le pthread_* non risolvono (#219). ARCH ?= native -CFLAGS = -O3 -march=$(ARCH) -fopenmp -Wall -Wextra -Wno-unused-parameter -Wno-misleading-indentation -Wno-unused-function -LDFLAGS = -lm -fopenmp +CFLAGS = -O3 -march=$(ARCH) -fopenmp -pthread -Wall -Wextra -Wno-unused-parameter -Wno-misleading-indentation -Wno-unused-function +LDFLAGS = -lm -fopenmp -pthread EXE = endif endif @@ -105,22 +125,51 @@ INSTALL ?= install # clean. See backend_loader.c and README "cuda-dll" below. CUDA ?= 0 CUDA_DLL ?= 0 +ifneq ($(IS_WIN),) +# the CUDA installer sets CUDA_PATH system-wide (e.g. C:\Program Files\NVIDIA +# GPU Computing Toolkit\CUDA\v13.2); fall back to it before the POSIX default. +# NVCC defaults to plain `nvcc` from PATH: CUDA_PATH contains spaces, which the +# unquoted recipe checks cannot survive — and the cuda-dll recipe already +# requires an MSVC environment (vcvars64) on PATH, so requiring the CUDA bin +# directory too is symmetric. The installer adds it by default. +CUDA_HOME ?= $(subst \,/,$(CUDA_PATH)) +NVCC ?= nvcc +# Host compiler override for nvcc (e.g. Fedora ships g++16 but CUDA needs 15): +# make glm CUDA=1 NVCC_CCBIN=g++-15 (opt-in: unset = nvcc's own default) +ifneq ($(NVCC_CCBIN),) +NVCCFLAGS += -ccbin $(NVCC_CCBIN) +endif +else CUDA_HOME ?= /usr/local/cuda NVCC ?= $(CUDA_HOME)/bin/nvcc +endif CUDA_ARCH ?= native -# -Xcompiler passes flags to nvcc's HOST compiler. On Windows that host is MSVC -# cl.exe, which rejects the gcc/clang-style -Wall,-Wextra (D8021). Gate them -# behind a non-MSVC host so the Linux/macOS (gcc/clang) build keeps the warnings -# and the Windows cuda-dll build doesn't fail. (#306) -NVCC_HOST_WARN ?= $(if $(IS_WIN),,-Xcompiler=-Wall,-Wextra) -NVCCFLAGS ?= -O3 -std=c++17 -arch=$(CUDA_ARCH) $(NVCC_HOST_WARN) -ifeq ($(IS_WIN),) -PYTHON ?= python3 +ifneq ($(IS_WIN),) +# nvcc's host compiler on Windows is MSVC cl.exe: the GCC-style +# -Xcompiler=-Wall,-Wextra is rejected ("D8021 invalid numeric argument +# '/Wextra'"), which made `make cuda-dll` unbuildable as shipped. -W3 is +# the MSVC warning level (dash form, NOT /W3: MSYS make would mangle the +# slash-form into a filesystem path). +NVCCFLAGS ?= -O3 -std=c++17 -arch=$(CUDA_ARCH) -Xcompiler=-W3 else +NVCCFLAGS ?= -O3 -std=c++17 -arch=$(CUDA_ARCH) -Xcompiler=-Wall,-Wextra +endif +# PYTHON is a HOST tool (clean/test-c/test-python run it on the build machine), +# so key it off the host, NOT $(IS_WIN) — which is derived from the *target* +# triple ($(CC) -dumpmachine). Otherwise a cross build (make CC=x86_64-w64- +# mingw32-gcc ... on Linux) sets IS_WIN and picks `python`, breaking clean/test +# on hosts where only `python3` exists. $(OS)=Windows_NT in every Windows shell +# (the #129 signal) and is empty on Linux/macOS. +ifeq ($(OS),Windows_NT) PYTHON ?= python +else +PYTHON ?= python3 endif CUDA_OBJ = -TEST_BINS = tests/test_json$(EXE) tests/test_st$(EXE) tests/test_tier$(EXE) tests/test_grammar$(EXE) tests/test_decode_batch$(EXE) tests/test_idot$(EXE) tests/test_i4_acc512$(EXE) tests/test_compat_direct$(EXE) +TEST_BINS = tests/test_json$(EXE) tests/test_st$(EXE) tests/test_tier$(EXE) tests/test_grammar$(EXE) tests/test_schema_gbnf$(EXE) tests/test_decode_batch$(EXE) tests/test_idot$(EXE) tests/test_kv_alloc$(EXE) tests/test_i4_acc512$(EXE) tests/test_compat_direct$(EXE) +ifneq (,$(LINUX)) +TEST_BINS += tests/test_uring$(EXE) +endif # Windows CUDA DLL path: host links the loader, NOT cudart. ifneq ($(IS_WIN),) @@ -148,7 +197,7 @@ endif # runtime, so no Xcode / offline metal compiler is required. Default build unchanged. METAL ?= 0 METAL_OBJ = -METALXX = clang++ -x objective-c++ -fobjc-arc -O3 +METALXX = clang++ -x objective-c++ -std=gnu++17 -fobjc-arc -O3 ifeq ($(METAL),1) ifeq (,$(DARWIN)) $(error METAL=1 is supported only on macOS) @@ -177,7 +226,7 @@ $(shell printf '%s' '$(BUILD_CONFIG)' > .build-config) endif .build-config: ; -glm$(EXE): glm.c st.h json.h tok.h tok_unicode.h compat.h grammar.h $(CUDA_OBJ) $(METAL_OBJ) .build-config +glm$(EXE): glm.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h $(CUDA_OBJ) $(METAL_OBJ) .build-config $(CC) $(CFLAGS) glm.c $(CUDA_OBJ) $(METAL_OBJ) -o glm$(EXE) $(LDFLAGS) # Windows runtime loader object: resolves coli_cuda_* from coli_cuda.dll. @@ -188,7 +237,7 @@ backend_loader.o: backend_loader.c backend_cuda.h compat.h .build-config # compiler, required by nvcc on Windows) into coli_cuda.dll. Run this from a # shell that has the MSVC environment set (e.g. after vcvars64.bat, or from a # "x64 Native Tools Command Prompt"). COLI_CUDA_BUILDING_DLL enables -# __declspec(dllexport) so the 11 API symbols are exported. +# __declspec(dllexport) so the 15 API symbols are exported. cuda-dll: backend_cuda.cu backend_cuda.h @command -v "$(NVCC)" >/dev/null 2>&1 || { echo "nvcc not found: set CUDA_HOME or NVCC" >&2; exit 1; } @command -v cl >/dev/null 2>&1 || { echo "cl.exe (MSVC) not in PATH — run vcvars64.bat first" >&2; exit 1; } @@ -220,9 +269,23 @@ cuda-bench: backend_cuda.cu backend_cuda.h tests/bench_tensor_core.cu olmoe$(EXE): olmoe.c st.h json.h compat.h $(CC) $(CFLAGS) olmoe.c -o olmoe$(EXE) $(LDFLAGS) -# binario portabile da distribuire su altre macchine x86-64 +# Use a baseline that matches the compiler target. macOS already targets a +# portable baseline when ARCH is empty; forcing the x86 value there breaks +# Apple Silicon. Unknown targets use native rather than an invalid x86 flag. +ifneq (,$(DARWIN)) +PORTABLE_ARCH = +else ifneq (,$(AARCH64)) +PORTABLE_ARCH = armv8-a +else ifneq (,$(PPC64)) +PORTABLE_ARCH = power8 +else ifneq (,$(X86_64)) +PORTABLE_ARCH = x86-64-v3 +else +PORTABLE_ARCH = native +endif + portable: - $(MAKE) glm$(EXE) ARCH=x86-64-v3 + $(MAKE) glm$(EXE) ARCH=$(PORTABLE_ARCH) iobench$(EXE): iobench.c compat.h $(CC) $(CFLAGS) iobench.c -o iobench$(EXE) $(LDFLAGS) @@ -239,10 +302,16 @@ tests/test_tier$(EXE): tests/test_tier.c tier.h tests/test_grammar$(EXE): tests/test_grammar.c grammar.h $(CC) $(CFLAGS) $< -o $@ $(LDFLAGS) +tests/test_schema_gbnf$(EXE): tests/test_schema_gbnf.c schema_gbnf.h grammar.h json.h + $(CC) $(CFLAGS) $< -o $@ $(LDFLAGS) + tests/test_decode_batch$(EXE): tests/test_decode_batch.c decode_batch.h $(CC) $(CFLAGS) $< -o $@ $(LDFLAGS) -tests/test_idot$(EXE): tests/test_idot.c glm.c st.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h +tests/test_idot$(EXE): tests/test_idot.c glm.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h + $(CC) $(CFLAGS) $< -o $@ $(LDFLAGS) + +tests/test_kv_alloc$(EXE): tests/test_kv_alloc.c glm.c st.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h $(CC) $(CFLAGS) $< -o $@ $(LDFLAGS) tests/test_i4_acc512$(EXE): tests/test_i4_acc512.c @@ -251,6 +320,9 @@ tests/test_i4_acc512$(EXE): tests/test_i4_acc512.c tests/test_compat_direct$(EXE): tests/test_compat_direct.c compat.h $(CC) $(CFLAGS) $< -o $@ $(LDFLAGS) +tests/test_uring$(EXE): tests/test_uring.c glm.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h + $(CC) $(CFLAGS) $< -o $@ $(LDFLAGS) + test-c: $(TEST_BINS) $(PYTHON) tools/run_tests.py $(TEST_BINS) @@ -284,4 +356,4 @@ clean: bench: iobench$(EXE) @if [ -n "$(ARGS)" ]; then ./iobench$(EXE) $(ARGS); else echo "built iobench$(EXE) — run: ./iobench$(EXE) "; fi -.PHONY: all glm cuda-test cuda-bench cuda-dll portable test-c test-python test check clean install uninstall bench \ No newline at end of file +.PHONY: all glm cuda-test cuda-bench cuda-dll portable test-c test-python test check clean install uninstall bench diff --git a/c/backend_cuda.cu b/c/backend_cuda.cu index d09a9fa..9ce142f 100644 --- a/c/backend_cuda.cu +++ b/c/backend_cuda.cu @@ -18,12 +18,14 @@ struct ColiCudaTensor { typedef struct { int device; + int compute_major,compute_minor; float *x, *y, *gate, *up; size_t x_cap, y_cap, gate_cap, up_cap; uint8_t *qx; float *qscale; size_t qx_cap, qscale_cap; float *host_x,*host_y; size_t host_x_cap,host_y_cap; float *aq,*al,*ar,*ac; size_t aq_cap,al_cap,ar_cap,ac_cap; + float *pipe_buf[24]; size_t pipe_cap[24]; /* scratch persistenti del resident pipeline */ cudaStream_t stream; void *group_desc; size_t group_desc_cap; size_t tensor_count, tensor_bytes; @@ -120,6 +122,68 @@ __global__ static void silu_mul(float *gate, const float *up, size_t n) { } } +/* Four warps share one A tile and compute 16x64 outputs. This matters for + * prefill: the first prototype reloaded/converter A once per 16 output cols. */ +__global__ static void w4a16_matmul(float *y,const float *x,const uint8_t *w, + const float *scale,int M,int K,int N){ +#if __CUDA_ARCH__ >= 700 + using namespace nvcuda;int warp=threadIdx.x>>5,lane=threadIdx.x&31; + int m0=blockIdx.y*16,n0=blockIdx.x*64+warp*16; + __shared__ __half ah[256],bh[4][256]; + wmma::fragment acc;wmma::fill_fragment(acc,0.f); + size_t rb=(size_t)(K+1)/2; + for(int k0=0;k0>1)];int a=(gk&1)?q>>4:q&15; + v=(float)(a&8?a-16:a)*scale[gn];} + bh[warp][z]=__float2half(v); /* [Ntile,Ktile] == B col-major */ + } + __syncthreads(); + wmma::fragment af; + wmma::fragment bf; + wmma::load_matrix_sync(af,ah,16);wmma::load_matrix_sync(bf,bh[warp],16); + wmma::mma_sync(acc,af,bf,acc);__syncthreads(); + } + __shared__ float out[4][256];wmma::store_matrix_sync(out[warp],acc,16,wmma::mem_row_major);__syncwarp(); + for(int z=lane;z<256;z+=32){int m=z/16,n=z%16; + if(m0+mFP16 conversion of A. */ +__global__ static void w4a16_gate_up(float *gate,float *up,const float *x, + const uint8_t *gw,const uint8_t *uw,const float *gs,const float *us, + int M,int K,int N){ +#if __CUDA_ARCH__ >= 700 + using namespace nvcuda;int warp=threadIdx.x>>5,lane=threadIdx.x&31,which=warp&1,tile=warp>>1; + int m0=blockIdx.y*16,n0=blockIdx.x*64+tile*16;const uint8_t *w=which?uw:gw; + const float *scale=which?us:gs;float *y=which?up:gate;size_t rb=(size_t)(K+1)/2; + __shared__ __half ah[256],bh[8][256]; + wmma::fragment acc;wmma::fill_fragment(acc,0.f); + for(int k0=0;k0>1)];int a=(gk&1)?q>>4:q&15; + v=(float)(a&8?a-16:a)*scale[gn];}bh[warp][z]=__float2half(v);} + __syncthreads(); + wmma::fragment af; + wmma::fragment bf; + wmma::load_matrix_sync(af,ah,16);wmma::load_matrix_sync(bf,bh[warp],16); + wmma::mma_sync(acc,af,bf,acc);__syncthreads(); + } + __shared__ float out[8][256];wmma::store_matrix_sync(out[warp],acc,16,wmma::mem_row_major);__syncwarp(); + for(int z=lane;z<256;z+=32){int m=z/16,n=z%16; + if(m0+m=S)return; const float *xs=x+(size_t)s*K; float v=0; for(int i=threadIdx.x;i=S||nt<1)return; + extern __shared__ float sm[];float *qa=sm,*cl=qa+K,*scores=cl+K,*red=scores+T; + const float *qs=q+((size_t)s*H+h)*(Q+R); + for(int k=tid;k>1;n;n>>=1){if(tid>1;n;n>>=1){if(tid= bytes) return 1; if (*ptr) cudaFree(*ptr); @@ -286,6 +381,7 @@ extern "C" int coli_cuda_init(const int *devices, int count) { if (!select_ctx(ctx)) { g_nctx = 0; return 0; } cudaDeviceProp prop{}; if (!cuda_ok(cudaGetDeviceProperties(&prop, device), "device properties")) { g_nctx = 0; return 0; } + ctx->compute_major=prop.major;ctx->compute_minor=prop.minor; if(!cuda_ok(cudaStreamCreateWithFlags(&ctx->stream,cudaStreamNonBlocking),"stream creation")){ g_nctx=0;return 0; } @@ -307,6 +403,7 @@ extern "C" void coli_cuda_shutdown(void) { if (ctx->qx) cudaFree(ctx->qx); if (ctx->qscale) cudaFree(ctx->qscale); if(ctx->aq)cudaFree(ctx->aq);if(ctx->al)cudaFree(ctx->al);if(ctx->ar)cudaFree(ctx->ar);if(ctx->ac)cudaFree(ctx->ac); + for(int b=0;b<24;b++) if(ctx->pipe_buf[b]) cudaFree(ctx->pipe_buf[b]); if (ctx->host_x) cudaFreeHost(ctx->host_x); if (ctx->host_y) cudaFreeHost(ctx->host_y); if (ctx->stream) cudaStreamDestroy(ctx->stream); @@ -388,6 +485,23 @@ extern "C" int coli_cuda_tensor_upload(ColiCudaTensor **tensor, return 1; } +extern "C" int coli_cuda_tensor_update(ColiCudaTensor *tensor, + const void *weights, + const float *scales) { + if (!tensor || !weights || (tensor->fmt && !scales)) return 0; + DeviceContext *ctx=find_ctx(tensor->device); + if (!select_ctx(ctx)) return 0; + if (!cuda_ok(cudaMemcpy(tensor->weights,weights,tensor->weight_bytes, + cudaMemcpyHostToDevice),"tensor refresh")) return 0; + if(tensor->fmt==2){ + offset_to_signed_s4<<<(unsigned)((tensor->weight_bytes+255)/256),256>>>( + (uint8_t*)tensor->weights,tensor->weight_bytes); + if(!cuda_ok(cudaGetLastError(),"int4 weight refresh")) return 0; + } + return !tensor->fmt || cuda_ok(cudaMemcpy(tensor->scales,scales, + (size_t)tensor->O*sizeof(float),cudaMemcpyHostToDevice),"scale refresh"); +} + extern "C" int coli_cuda_matmul(ColiCudaTensor **tensor, float *y, const float *x, const void *weights, const float *scales, @@ -436,6 +550,34 @@ extern "C" int coli_cuda_expert_mlp(ColiCudaTensor *gate, ColiCudaTensor *up, return 1; } +extern "C" int coli_cuda_shared_mlp_w4a16(ColiCudaTensor *gate,ColiCudaTensor *up, + ColiCudaTensor *down,float *y,const float *x,int S){ + if(!gate||!up||!down||!x||!y||S<1||gate->fmt!=2||up->fmt!=2||down->fmt!=2|| + gate->device!=up->device||gate->device!=down->device||gate->I!=up->I|| + gate->O!=up->O||down->I!=gate->O||down->O!=gate->I)return 0; + DeviceContext *ctx=find_ctx(gate->device);if(!select_ctx(ctx)||ctx->compute_major<7)return 0; + int D=gate->I,I=gate->O;size_t xb=(size_t)S*D*sizeof(float),ib=(size_t)S*I*sizeof(float); + if(!reserve(&ctx->x,&ctx->x_cap,xb)||!reserve(&ctx->gate,&ctx->gate_cap,ib)|| + !reserve(&ctx->up,&ctx->up_cap,ib)||!reserve(&ctx->y,&ctx->y_cap,xb)|| + !reserve_pinned(&ctx->host_x,&ctx->host_x_cap,xb)|| + !reserve_pinned(&ctx->host_y,&ctx->host_y_cap,xb))return 0; + std::memcpy(ctx->host_x,x,xb); + if(!cuda_ok(cudaMemcpyAsync(ctx->x,ctx->host_x,xb,cudaMemcpyHostToDevice,ctx->stream), + "shared w4a16 input upload"))return 0; + dim3 hidden((unsigned)((I+63)/64),(unsigned)((S+15)/16)); + dim3 output((unsigned)((D+63)/64),(unsigned)((S+15)/16)); + w4a16_gate_up<<stream>>>(ctx->gate,ctx->up,ctx->x, + (const uint8_t*)gate->weights,(const uint8_t*)up->weights,gate->scales,up->scales,S,D,I); + silu_mul<<<(unsigned)(((size_t)S*I+255)/256),256,0,ctx->stream>>>(ctx->gate,ctx->up,(size_t)S*I); + w4a16_matmul<<stream>>>(ctx->y,ctx->gate,(const uint8_t*)down->weights,down->scales,S,I,D); + if(!cuda_ok(cudaGetLastError(),"shared w4a16 launch")|| + !cuda_ok(cudaMemcpyAsync(ctx->host_y,ctx->y,xb,cudaMemcpyDeviceToHost,ctx->stream), + "shared w4a16 output download")|| + !cuda_ok(cudaStreamSynchronize(ctx->stream),"shared w4a16 synchronize"))return 0; + std::memcpy(y,ctx->host_y,xb); + return 1; +} + extern "C" int coli_cuda_expert_group(ColiCudaTensor *const *gates, ColiCudaTensor *const *ups, ColiCudaTensor *const *downs, @@ -493,6 +635,38 @@ extern "C" int coli_cuda_expert_group(ColiCudaTensor *const *gates, silu_mul<<<(unsigned)(((size_t)total*I+255)/256),256,0,ctx->stream>>>(ctx->gate,ctx->up,(size_t)total*I); quantize_s4_rows<<stream>>>(ctx->qx,ctx->qscale,ctx->gate,total,I); grouped_s4_wmma<<stream>>>(ctx->y,ctx->qx,ctx->qscale,dev,I,D,2); + }else if(all_s4&&ctx->compute_major>=7&&getenv("COLI_CUDA_TC_W4A16")&& + atoi(getenv("COLI_CUDA_TC_W4A16"))){ + /* W4A16 Tensor Core per gruppo: attivazioni fp16 per tile (lossless al + * contrario del path W4A4), un lancio per expert dentro lo stream — + * l'overhead di lancio e' trascurabile rispetto ai GEMM. */ + int tc16_min=getenv("COLI_CUDA_TC_W4A16_MIN")?atoi(getenv("COLI_CUDA_TC_W4A16_MIN")):16; + int off16=0; + for(int c=0;cgate+(size_t)off16*I,*u16=ctx->up+(size_t)off16*I; + float *x16=ctx->x+(size_t)off16*D,*y16=ctx->y+(size_t)off16*D; + if(r>=tc16_min){ + dim3 hg16((unsigned)((I+63)/64),(unsigned)((r+15)/16)); + dim3 og16((unsigned)((D+63)/64),(unsigned)((r+15)/16)); + w4a16_gate_up<<stream>>>(g16,u16,x16, + (const uint8_t*)host[c].g,(const uint8_t*)host[c].u,host[c].gs,host[c].us,r,D,I); + silu_mul<<<(unsigned)(((size_t)r*I+255)/256),256,0,ctx->stream>>>(g16,u16,(size_t)r*I); + w4a16_matmul<<stream>>>(y16,g16, + (const uint8_t*)host[c].d,host[c].ds,r,I,D); + }else{ + /* piccoli batch: tile TC quasi vuoti + overhead di lancio — il + * kernel naive per-elemento resta piu' veloce (misurato in decode) */ + quant_matmul<<stream>>>(g16,x16, + host[c].g,host[c].gs,host[c].gf,r,D,I,row_bytes(host[c].gf,D)); + quant_matmul<<stream>>>(u16,x16, + host[c].u,host[c].us,host[c].uf,r,D,I,row_bytes(host[c].uf,D)); + silu_mul<<<(unsigned)(((size_t)r*I+255)/256),256,0,ctx->stream>>>(g16,u16,(size_t)r*I); + quant_matmul<<stream>>>(y16,g16, + host[c].d,host[c].ds,host[c].df,r,I,D,row_bytes(host[c].df,I)); + } + off16+=r; + } }else if(all_s4&&(!getenv("COLI_CUDA_W4_PACKED")||atoi(getenv("COLI_CUDA_W4_PACKED")))){ dim3 hg((unsigned)I,(unsigned)max_rows,(unsigned)count),og((unsigned)D,(unsigned)max_rows,(unsigned)count); int dual=!getenv("COLI_CUDA_DUAL_PROJ")||atoi(getenv("COLI_CUDA_DUAL_PROJ")); @@ -530,6 +704,7 @@ extern "C" int coli_cuda_expert_group(ColiCudaTensor *const *gates, return 1; } + extern "C" int coli_cuda_attention_absorb(ColiCudaTensor *w,float *ctx,const float *q, const float *latent,const float *rope,int H,int Q, int R,int V,int K,int T,float scale){ @@ -552,6 +727,49 @@ extern "C" int coli_cuda_attention_absorb(ColiCudaTensor *w,float *ctx,const flo return 1; } +static int attention_absorb_batch_run(ColiCudaTensor *w,ColiCudaTensor *proj,float *out, + const float *q,const float *latent,const float *rope,int S,int H,int Q,int R,int V, + int K,int T,float scale){ + if(!w||!out||!q||!latent||!rope||S<1||H<1||Q<1||R<1||V<1||K<1||K>512|| + T8192||w->I!=K||w->O!=H*(Q+V))return 0; + if(proj&&(proj->device!=w->device||proj->I!=H*V))return 0; + DeviceContext *dc=find_ctx(w->device);if(!select_ctx(dc))return 0; + size_t qb=(size_t)S*H*(Q+R)*sizeof(float),lb=(size_t)T*K*sizeof(float); + size_t rb=(size_t)T*R*sizeof(float),cb=(size_t)S*H*V*sizeof(float); + if(!reserve(&dc->aq,&dc->aq_cap,qb)||!reserve(&dc->al,&dc->al_cap,lb)|| + !reserve(&dc->ar,&dc->ar_cap,rb)||!reserve(&dc->ac,&dc->ac_cap,cb))return 0; + if(!cuda_ok(cudaMemcpyAsync(dc->aq,q,qb,cudaMemcpyHostToDevice,dc->stream),"attention batch q upload")|| + !cuda_ok(cudaMemcpyAsync(dc->al,latent,lb,cudaMemcpyHostToDevice,dc->stream),"attention batch latent upload")|| + !cuda_ok(cudaMemcpyAsync(dc->ar,rope,rb,cudaMemcpyHostToDevice,dc->stream),"attention batch rope upload"))return 0; + size_t shared=(size_t)(2*K+T+256)*sizeof(float); + attention_absorb_batch_kernel<<stream>>>(dc->ac,dc->aq,dc->al, + dc->ar,w->weights,w->scales,w->fmt,S,H,Q,R,V,K,T,scale); + if(!cuda_ok(cudaGetLastError(),"attention batch launch"))return 0; + const float *src=dc->ac;size_t ob=cb; + if(proj){ + ob=(size_t)S*proj->O*sizeof(float);if(!reserve(&dc->y,&dc->y_cap,ob))return 0; + quant_matmul<<O,S),256,0,dc->stream>>>(dc->y,dc->ac,proj->weights, + proj->scales,proj->fmt,S,proj->I,proj->O,row_bytes(proj->fmt,proj->I)); + if(!cuda_ok(cudaGetLastError(),"attention o_proj launch"))return 0;src=dc->y; + } + if(!cuda_ok(cudaMemcpyAsync(out,src,ob,cudaMemcpyDeviceToHost,dc->stream), + proj?"attention projected output download":"attention batch context download")|| + !cuda_ok(cudaStreamSynchronize(dc->stream),"attention batch synchronize"))return 0; + return 1; +} + +extern "C" int coli_cuda_attention_absorb_batch(ColiCudaTensor *w,float *ctx,const float *q, + const float *latent,const float *rope,int S,int H,int Q,int R,int V,int K,int T, + float scale){ + return attention_absorb_batch_run(w,nullptr,ctx,q,latent,rope,S,H,Q,R,V,K,T,scale); +} + +extern "C" int coli_cuda_attention_project_batch(ColiCudaTensor *w,ColiCudaTensor *proj, + float *out,const float *q,const float *latent,const float *rope,int S,int H,int Q, + int R,int V,int K,int T,float scale){ + return attention_absorb_batch_run(w,proj,out,q,latent,rope,S,H,Q,R,V,K,T,scale); +} + extern "C" void coli_cuda_tensor_free(ColiCudaTensor *tensor) { if (!tensor) return; DeviceContext *ctx = find_ctx(tensor->device); @@ -573,3 +791,234 @@ extern "C" size_t coli_cuda_tensor_bytes(const ColiCudaTensor *tensor) { extern "C" int coli_cuda_tensor_device(const ColiCudaTensor *tensor) { return tensor ? tensor->device : -1; } + +/* ==== resident-pipeline primitives (Inc.0, 2026-07-13) ==== + * Device-side building blocks so the residual stream can stay on the layer's + * home device across a whole layer. Control flow stays on CPU; only the data + * plane lives here. All entry points take DEVICE pointers (no transfers) — + * the caller owns staging via the pipe buffer API below. */ + +__global__ static void pipe_rmsnorm_rows(float *y,const float *x,const float *w, + int D,float eps,int xstride,int ystride){ + const float *xr=x+(size_t)blockIdx.x*xstride; float *yr=y+(size_t)blockIdx.x*ystride; + __shared__ double sh[256]; + double a=0; for(int i=threadIdx.x;i0;s>>=1){ if(threadIdx.x=24||!select_ctx(ctx)) return NULL; + if(!reserve(&ctx->pipe_buf[slot],&ctx->pipe_cap[slot],bytes)) return NULL; + return ctx->pipe_buf[slot]; +} +extern "C" void *coli_cuda_pipe_alloc(int device,size_t bytes){ + DeviceContext *ctx=find_ctx(device); if(!select_ctx(ctx)) return NULL; + void *p=NULL; + if(!cuda_ok(cudaMalloc(&p,bytes),"pipe alloc")) return NULL; + return p; +} +extern "C" void coli_cuda_pipe_free(int device,void *p){ + DeviceContext *ctx=find_ctx(device); if(!p||!select_ctx(ctx)) return; + cudaFree(p); +} +extern "C" int coli_cuda_pipe_upload(int device,void *dst,const void *src,size_t bytes){ + DeviceContext *ctx=find_ctx(device); if(!select_ctx(ctx)) return 0; + return cuda_ok(cudaMemcpy(dst,src,bytes,cudaMemcpyHostToDevice),"pipe upload"); +} +extern "C" int coli_cuda_pipe_download(int device,const void *src,void *dst,size_t bytes){ + DeviceContext *ctx=find_ctx(device); if(!select_ctx(ctx)) return 0; + return cuda_ok(cudaMemcpy(dst,src,bytes,cudaMemcpyDeviceToHost),"pipe download"); +} +extern "C" int coli_cuda_pipe_rmsnorm(int device,float *y_dev,const float *x_dev, + const float *w_dev,int S,int D,float eps){ + DeviceContext *ctx=find_ctx(device); + if(S<1||D<1||!select_ctx(ctx)) return 0; + pipe_rmsnorm_rows<<>>(y_dev,x_dev,w_dev,D,eps,D,D); + return cuda_ok(cudaGetLastError(),"pipe rmsnorm"); +} +extern "C" int coli_cuda_pipe_rmsnorm_s(int device,float *y_dev,const float *x_dev, + const float *w_dev,int S,int D,float eps, + int xstride,int ystride){ + DeviceContext *ctx=find_ctx(device); + if(S<1||D<1||xstride>>(y_dev,x_dev,w_dev,D,eps,xstride,ystride); + return cuda_ok(cudaGetLastError(),"pipe rmsnorm strided"); +} +extern "C" int coli_cuda_pipe_rope(int device,float *v_dev,const int *pos_dev, + int rows,int stride,int offset,int R,int heads, + float theta){ + DeviceContext *ctx=find_ctx(device); + if(rows<1||R<2||R>256||heads<1||!select_ctx(ctx)) return 0; + pipe_rope_rows<<>>(v_dev,pos_dev,0,stride,offset,R,heads,theta); + return cuda_ok(cudaGetLastError(),"pipe rope"); +} +extern "C" int coli_cuda_pipe_rope_base(int device,float *v_dev,int pos_base,int rows, + int stride,int offset,int R,int heads,float theta){ + DeviceContext *ctx=find_ctx(device); + if(rows<1||R<2||R>256||heads<1||!select_ctx(ctx)) return 0; + pipe_rope_rows<<>>(v_dev,NULL,pos_base,stride,offset,R,heads,theta); + return cuda_ok(cudaGetLastError(),"pipe rope base"); +} +extern "C" int coli_cuda_pipe_copy2d(int device,float *dst,int dpitch,const float *src, + int spitch,int width,int height){ + DeviceContext *ctx=find_ctx(device); if(!select_ctx(ctx)) return 0; + return cuda_ok(cudaMemcpy2D(dst,(size_t)dpitch*4,src,(size_t)spitch*4, + (size_t)width*4,height,cudaMemcpyDeviceToDevice),"pipe copy2d"); +} +/* attention batch + fused o_proj with DEVICE-resident q/latent/rope: the whole + * upstream projection chain stayed on this device, so nothing is uploaded here. + * Only the final [S,O] projection is downloaded to host. */ +extern "C" int coli_cuda_attention_project_batch_dev(ColiCudaTensor *w,ColiCudaTensor *proj, + float *out,const float *q_dev,const float *latent_dev,const float *rope_dev, + int S,int H,int Q,int R,int V,int K,int T,float scale){ + if(!w||!proj||!out||!q_dev||!latent_dev||!rope_dev||S<1||H<1||Q<1||R<1||V<1|| + K<1||K>512||T8192||w->I!=K||w->O!=H*(Q+V)|| + proj->device!=w->device||proj->I!=H*V)return 0; + DeviceContext *dc=find_ctx(w->device);if(!select_ctx(dc))return 0; + size_t cb=(size_t)S*H*V*sizeof(float); + if(!reserve(&dc->ac,&dc->ac_cap,cb))return 0; + size_t shared=(size_t)(2*K+T+256)*sizeof(float); + attention_absorb_batch_kernel<<stream>>>(dc->ac,q_dev,latent_dev, + rope_dev,w->weights,w->scales,w->fmt,S,H,Q,R,V,K,T,scale); + if(!cuda_ok(cudaGetLastError(),"pipe attention launch"))return 0; + size_t ob=(size_t)S*proj->O*sizeof(float); + if(!reserve(&dc->y,&dc->y_cap,ob))return 0; + quant_matmul<<O,S),256,0,dc->stream>>>(dc->y,dc->ac,proj->weights, + proj->scales,proj->fmt,S,proj->I,proj->O,row_bytes(proj->fmt,proj->I)); + if(!cuda_ok(cudaGetLastError(),"pipe o_proj launch"))return 0; + if(!cuda_ok(cudaMemcpyAsync(out,dc->y,ob,cudaMemcpyDeviceToHost,dc->stream),"pipe attention download")|| + !cuda_ok(cudaStreamSynchronize(dc->stream),"pipe attention sync"))return 0; + return 1; +} +extern "C" int coli_cuda_pipe_silu_mul(int device,float *gate_dev,const float *up_dev, + size_t n){ + DeviceContext *ctx=find_ctx(device); if(!n||!select_ctx(ctx)) return 0; + silu_mul<<<(unsigned)((n+255)/256),256>>>(gate_dev,up_dev,n); + return cuda_ok(cudaGetLastError(),"pipe silu mul"); +} +extern "C" int coli_cuda_pipe_add(int device,float *x_dev,const float *t_dev,size_t n){ + DeviceContext *ctx=find_ctx(device); if(!n||!select_ctx(ctx)) return 0; + pipe_add_n<<<(unsigned)((n+255)/256),256>>>(x_dev,t_dev,n); + return cuda_ok(cudaGetLastError(),"pipe add"); +} +extern "C" int coli_cuda_pipe_rows_add(int device,float *x_dev,const float *partial_dev, + const int *rows_dev,int nrows,int D){ + DeviceContext *ctx=find_ctx(device); if(nrows<1||D<1||!select_ctx(ctx)) return 0; + pipe_rows_add<<>>(x_dev,partial_dev,rows_dev,D); + return cuda_ok(cudaGetLastError(),"pipe rows add"); +} +/* GEMM with device-resident activations: same quant_matmul kernel as + * coli_cuda_matmul, zero host transfers. */ +extern "C" int coli_cuda_pipe_gemm(ColiCudaTensor *t,float *y_dev,const float *x_dev, + int S){ + if(!t||S<1) return 0; + DeviceContext *ctx=find_ctx(t->device); if(!select_ctx(ctx)) return 0; + dim3 grid((unsigned)t->O,(unsigned)S); + quant_matmul<<>>(y_dev,x_dev,t->weights,t->scales,t->fmt,S,t->I,t->O, + row_bytes(t->fmt,t->I)); + return cuda_ok(cudaGetLastError(),"pipe gemm"); +} +/* copia diretta scheda->scheda (P2P se disponibile, altrimenti staging driver) */ +extern "C" int coli_cuda_pipe_peer_copy(int dst_dev,float *dst,int src_dev, + const float *src,size_t bytes){ + if(!dst||!src) return 0; + if(dst_dev==src_dev){ DeviceContext *c=find_ctx(dst_dev); if(!select_ctx(c)) return 0; + return cuda_ok(cudaMemcpy(dst,src,bytes,cudaMemcpyDeviceToDevice),"pipe intra copy"); } + return cuda_ok(cudaMemcpyPeer(dst,dst_dev,src,src_dev,bytes),"pipe peer copy"); +} +/* come attention_project_batch_dev ma l'uscita di o_proj RESTA sul device (out_dev). */ +extern "C" int coli_cuda_attention_project_batch_dev_out(ColiCudaTensor *w,ColiCudaTensor *proj, + float *out_dev,const float *q_dev,const float *latent_dev,const float *rope_dev, + int S,int H,int Q,int R,int V,int K,int T,float scale){ + if(!w||!proj||!out_dev||!q_dev||!latent_dev||!rope_dev||S<1||H<1||Q<1||R<1||V<1|| + K<1||K>512||T8192||w->I!=K||w->O!=H*(Q+V)|| + proj->device!=w->device||proj->I!=H*V)return 0; + DeviceContext *dc=find_ctx(w->device);if(!select_ctx(dc))return 0; + size_t cb=(size_t)S*H*V*sizeof(float); + if(!reserve(&dc->ac,&dc->ac_cap,cb))return 0; + size_t shared=(size_t)(2*K+T+256)*sizeof(float); + attention_absorb_batch_kernel<<stream>>>(dc->ac,q_dev,latent_dev, + rope_dev,w->weights,w->scales,w->fmt,S,H,Q,R,V,K,T,scale); + if(!cuda_ok(cudaGetLastError(),"pipe attention launch (dev out)"))return 0; + quant_matmul<<O,S),256,0,dc->stream>>>(out_dev,dc->ac,proj->weights, + proj->scales,proj->fmt,S,proj->I,proj->O,row_bytes(proj->fmt,proj->I)); + if(!cuda_ok(cudaGetLastError(),"pipe o_proj launch (dev out)"))return 0; + return cuda_ok(cudaStreamSynchronize(dc->stream),"pipe attention sync (dev out)"); +} +/* absorb batch con TUTTO su device (q/latent/rope gia' residenti sulla scheda + * dello shard, ctx resta sul device): il cuore della attention head-shardata + * dentro il pipeline. Nessun trasferimento host. */ +extern "C" int coli_cuda_attention_absorb_batch_dev(ColiCudaTensor *w,float *ctx_dev, + const float *q_dev,const float *latent_dev,const float *rope_dev, + int S,int H,int Q,int R,int V,int K,int T,float scale){ + if(!w||!ctx_dev||!q_dev||!latent_dev||!rope_dev||S<1||H<1||Q<1||R<1||V<1|| + K<1||K>512||T8192||w->I!=K||w->O!=H*(Q+V))return 0; + DeviceContext *dc=find_ctx(w->device);if(!select_ctx(dc))return 0; + size_t shared=(size_t)(2*K+T+256)*sizeof(float); + attention_absorb_batch_kernel<<stream>>>(ctx_dev,q_dev,latent_dev, + rope_dev,w->weights,w->scales,w->fmt,S,H,Q,R,V,K,T,scale); + if(!cuda_ok(cudaGetLastError(),"pipe shard attention launch"))return 0; + return cuda_ok(cudaStreamSynchronize(dc->stream),"pipe shard attention sync"); +} +/* absorb per il DECODE con KV gia' residente: carica solo q (poche KB), + * latent/rope arrivano dall'ombra device. ctx torna a host (S piccolo). */ +extern "C" int coli_cuda_attention_absorb_kvdev(ColiCudaTensor *w,float *ctx,const float *q, + const float *latent_dev,const float *rope_dev,int H,int Q,int R,int V,int K,int T, + float scale){ + if(!w||!ctx||!q||!latent_dev||!rope_dev||H<1||Q<1||R<1||V<1||K<1||K>512||T<1||T>8192|| + w->I!=K||w->O!=H*(Q+V))return 0; + DeviceContext *dc=find_ctx(w->device);if(!select_ctx(dc))return 0; + size_t qb=(size_t)H*(Q+R)*sizeof(float),cb=(size_t)H*V*sizeof(float); + if(!reserve(&dc->aq,&dc->aq_cap,qb)||!reserve(&dc->ac,&dc->ac_cap,cb))return 0; + if(!cuda_ok(cudaMemcpyAsync(dc->aq,q,qb,cudaMemcpyHostToDevice,dc->stream),"kvdev q upload"))return 0; + size_t shared=(size_t)(2*K+T+256)*sizeof(float); + attention_absorb_batch_kernel<<stream>>>(dc->ac,dc->aq,latent_dev, + rope_dev,w->weights,w->scales,w->fmt,1,H,Q,R,V,K,T,scale); + if(!cuda_ok(cudaGetLastError(),"kvdev absorb launch")|| + !cuda_ok(cudaMemcpyAsync(ctx,dc->ac,cb,cudaMemcpyDeviceToHost,dc->stream),"kvdev ctx download")|| + !cuda_ok(cudaStreamSynchronize(dc->stream),"kvdev absorb sync"))return 0; + return 1; +} +extern "C" int coli_cuda_pipe_sync(int device){ + DeviceContext *ctx=find_ctx(device); if(!select_ctx(ctx)) return 0; + return cuda_ok(cudaDeviceSynchronize(),"pipe sync"); +} diff --git a/c/backend_cuda.h b/c/backend_cuda.h index 8112555..acbe4bc 100644 --- a/c/backend_cuda.h +++ b/c/backend_cuda.h @@ -56,6 +56,14 @@ COLI_CUDA_DLLEXPORT int coli_cuda_matmul(ColiCudaTensor **tensor, COLI_CUDA_DLLEXPORT int coli_cuda_expert_mlp(ColiCudaTensor *gate, ColiCudaTensor *up, ColiCudaTensor *down, float *y, const float *x, int S); +/* Prefill-oriented shared expert path. INT4 weights stay packed in global + * memory, activations are converted to FP16 per tile, and Tensor Cores + * accumulate into FP32. Unlike COLI_CUDA_TC_INT4 this does not quantize the + * activation to INT4. */ +COLI_CUDA_DLLEXPORT int coli_cuda_shared_mlp_w4a16(ColiCudaTensor *gate, ColiCudaTensor *up, + ColiCudaTensor *down, float *y, + const float *x, int S); + /* Packed group of same-shaped experts. Inputs and outputs contain sum(rows) * consecutive [D] rows in call order. */ COLI_CUDA_DLLEXPORT int coli_cuda_expert_group(ColiCudaTensor *const *gates, @@ -69,12 +77,70 @@ COLI_CUDA_DLLEXPORT int coli_cuda_attention_absorb(ColiCudaTensor *kv_b,float *c const float *latent,const float *rope,int H,int Q, int R,int V,int K,int T,float attention_scale); +/* Causal MLA absorption for S contiguous rows from one sequence. The KV + * arrays contain T rows ending at the final query; query s attends T-S+s+1 + * rows. One transfer and one launch replace S host round-trips. */ +COLI_CUDA_DLLEXPORT int coli_cuda_attention_absorb_batch(ColiCudaTensor *kv_b,float *ctx,const float *q, + const float *latent,const float *rope,int S, + int H,int Q,int R,int V,int K,int T, + float attention_scale); + +/* Same attention batch followed immediately by resident o_proj on the same + * device. Only the final [S,D] tensor crosses back to the host. */ +COLI_CUDA_DLLEXPORT int coli_cuda_attention_project_batch(ColiCudaTensor *kv_b,ColiCudaTensor *o_proj, + float *out,const float *q,const float *latent, + const float *rope,int S,int H,int Q,int R, + int V,int K,int T,float attention_scale); + COLI_CUDA_DLLEXPORT void coli_cuda_tensor_free(ColiCudaTensor *tensor); COLI_CUDA_DLLEXPORT size_t coli_cuda_tensor_bytes(const ColiCudaTensor *tensor); COLI_CUDA_DLLEXPORT int coli_cuda_tensor_device(const ColiCudaTensor *tensor); +/* Replace a resident tensor's contents without reallocating its device slot. */ +COLI_CUDA_DLLEXPORT int coli_cuda_tensor_update(ColiCudaTensor *tensor, + const void *weights, const float *scales); + +/* ---- resident-pipeline primitives (Inc.0): device-pointer entry points ---- */ +COLI_CUDA_DLLEXPORT float *coli_cuda_pipe_scratch(int device,int slot,size_t bytes); +COLI_CUDA_DLLEXPORT void *coli_cuda_pipe_alloc(int device,size_t bytes); +COLI_CUDA_DLLEXPORT void coli_cuda_pipe_free(int device,void *p); +COLI_CUDA_DLLEXPORT int coli_cuda_pipe_upload(int device,void *dst,const void *src,size_t bytes); +COLI_CUDA_DLLEXPORT int coli_cuda_pipe_download(int device,const void *src,void *dst,size_t bytes); +COLI_CUDA_DLLEXPORT int coli_cuda_pipe_rmsnorm(int device,float *y_dev,const float *x_dev, + const float *w_dev,int S,int D,float eps); +COLI_CUDA_DLLEXPORT int coli_cuda_pipe_rope(int device,float *v_dev,const int *pos_dev,int rows, + int stride,int offset,int R,int heads,float theta); +COLI_CUDA_DLLEXPORT int coli_cuda_pipe_silu_mul(int device,float *gate_dev,const float *up_dev,size_t n); +COLI_CUDA_DLLEXPORT int coli_cuda_pipe_add(int device,float *x_dev,const float *t_dev,size_t n); +COLI_CUDA_DLLEXPORT int coli_cuda_pipe_rows_add(int device,float *x_dev,const float *partial_dev, + const int *rows_dev,int nrows,int D); +COLI_CUDA_DLLEXPORT int coli_cuda_pipe_gemm(ColiCudaTensor *t,float *y_dev,const float *x_dev,int S); +COLI_CUDA_DLLEXPORT int coli_cuda_pipe_rmsnorm_s(int device,float *y_dev,const float *x_dev, + const float *w_dev,int S,int D,float eps, + int xstride,int ystride); +COLI_CUDA_DLLEXPORT int coli_cuda_pipe_rope_base(int device,float *v_dev,int pos_base,int rows, + int stride,int offset,int R,int heads,float theta); +COLI_CUDA_DLLEXPORT int coli_cuda_pipe_copy2d(int device,float *dst,int dpitch,const float *src, + int spitch,int width,int height); +COLI_CUDA_DLLEXPORT int coli_cuda_attention_project_batch_dev(ColiCudaTensor *kv_b,ColiCudaTensor *o_proj, + float *out,const float *q_dev,const float *latent_dev,const float *rope_dev, + int S,int H,int Q,int R,int V,int K,int T,float scale); +COLI_CUDA_DLLEXPORT int coli_cuda_attention_absorb_batch_dev(ColiCudaTensor *kv_b_shard,float *ctx_dev, + const float *q_dev,const float *latent_dev,const float *rope_dev, + int S,int H,int Q,int R,int V,int K,int T,float scale); +COLI_CUDA_DLLEXPORT int coli_cuda_attention_absorb_kvdev(ColiCudaTensor *kv_b,float *ctx,const float *q, + const float *latent_dev,const float *rope_dev,int H,int Q,int R,int V,int K,int T, + float scale); +COLI_CUDA_DLLEXPORT int coli_cuda_pipe_peer_copy(int dst_dev,float *dst,int src_dev, + const float *src,size_t bytes); +COLI_CUDA_DLLEXPORT int coli_cuda_attention_project_batch_dev_out(ColiCudaTensor *kv_b,ColiCudaTensor *o_proj, + float *out_dev,const float *q_dev,const float *latent_dev,const float *rope_dev, + int S,int H,int Q,int R,int V,int K,int T,float scale); +COLI_CUDA_DLLEXPORT int coli_cuda_pipe_sync(int device); + #ifdef __cplusplus } #endif #endif + diff --git a/c/backend_loader.c b/c/backend_loader.c index 6fea70c..b743d1b 100644 --- a/c/backend_loader.c +++ b/c/backend_loader.c @@ -22,6 +22,7 @@ #include #include +#include #include #include "backend_cuda.h" @@ -52,6 +53,35 @@ typedef void (*fn_tensor_free)(ColiCudaTensor *tensor); typedef size_t (*fn_tensor_bytes)(const ColiCudaTensor *tensor); typedef int (*fn_tensor_device)(const ColiCudaTensor *tensor); +/* --- #111 GPU resident pipeline additions (matched to backend_cuda.h) --- */ + + +/* --- #111 GPU resident pipeline additions (matched to backend_cuda.h) --- */ +typedef int (*fn_attention_absorb_batch)(ColiCudaTensor *kv_b,float *ctx,const float *q, const float *latent,const float *rope,int S, int H,int Q,int R,int V,int K,int T, float attention_scale); +typedef int (*fn_attention_absorb_batch_dev)(ColiCudaTensor *kv_b_shard,float *ctx_dev, const float *q_dev,const float *latent_dev,const float *rope_dev, int S,int H,int Q,int R,int V,int K,int T,float scale); +typedef int (*fn_attention_absorb_kvdev)(ColiCudaTensor *kv_b,float *ctx,const float *q, const float *latent_dev,const float *rope_dev,int H,int Q,int R,int V,int K,int T, float scale); +typedef int (*fn_attention_project_batch)(ColiCudaTensor *kv_b,ColiCudaTensor *o_proj, float *out,const float *q,const float *latent, const float *rope,int S,int H,int Q,int R, int V,int K,int T,float attention_scale); +typedef int (*fn_attention_project_batch_dev)(ColiCudaTensor *kv_b,ColiCudaTensor *o_proj, float *out,const float *q_dev,const float *latent_dev,const float *rope_dev, int S,int H,int Q,int R,int V,int K,int T,float scale); +typedef int (*fn_attention_project_batch_dev_out)(ColiCudaTensor *kv_b,ColiCudaTensor *o_proj, float *out_dev,const float *q_dev,const float *latent_dev,const float *rope_dev, int S,int H,int Q,int R,int V,int K,int T,float scale); +typedef int (*fn_pipe_add)(int device,float *x_dev,const float *t_dev,size_t n); +typedef void * (*fn_pipe_alloc)(int device,size_t bytes); +typedef int (*fn_pipe_copy2d)(int device,float *dst,int dpitch,const float *src, int spitch,int width,int height); +typedef int (*fn_pipe_download)(int device,const void *src,void *dst,size_t bytes); +typedef void (*fn_pipe_free)(int device,void *p); +typedef int (*fn_pipe_gemm)(ColiCudaTensor *t,float *y_dev,const float *x_dev,int S); +typedef int (*fn_pipe_peer_copy)(int dst_dev,float *dst,int src_dev, const float *src,size_t bytes); +typedef int (*fn_pipe_rmsnorm)(int device,float *y_dev,const float *x_dev, const float *w_dev,int S,int D,float eps); +typedef int (*fn_pipe_rmsnorm_s)(int device,float *y_dev,const float *x_dev, const float *w_dev,int S,int D,float eps, int xstride,int ystride); +typedef int (*fn_pipe_rope)(int device,float *v_dev,const int *pos_dev,int rows, int stride,int offset,int R,int heads,float theta); +typedef int (*fn_pipe_rope_base)(int device,float *v_dev,int pos_base,int rows, int stride,int offset,int R,int heads,float theta); +typedef int (*fn_pipe_rows_add)(int device,float *x_dev,const float *partial_dev, const int *rows_dev,int nrows,int D); +typedef float * (*fn_pipe_scratch)(int device,int slot,size_t bytes); +typedef int (*fn_pipe_silu_mul)(int device,float *gate_dev,const float *up_dev,size_t n); +typedef int (*fn_pipe_sync)(int device); +typedef int (*fn_pipe_upload)(int device,void *dst,const void *src,size_t bytes); +typedef int (*fn_shared_mlp_w4a16)(ColiCudaTensor *gate, ColiCudaTensor *up, ColiCudaTensor *down, float *y, const float *x, int S); +typedef int (*fn_tensor_update)(ColiCudaTensor *tensor, const void *weights, const float *scales); + /* Resolved pointers, plus a flag so we attempt the load at most once. */ static struct { int loaded; /* 1 = load attempted (success or fail), 0 = not yet */ @@ -72,6 +102,31 @@ static struct { fn_tensor_free tensor_free; fn_tensor_bytes tensor_bytes; fn_tensor_device tensor_device; + + fn_attention_absorb_batch attention_absorb_batch; + fn_attention_absorb_batch_dev attention_absorb_batch_dev; + fn_attention_absorb_kvdev attention_absorb_kvdev; + fn_attention_project_batch attention_project_batch; + fn_attention_project_batch_dev attention_project_batch_dev; + fn_attention_project_batch_dev_out attention_project_batch_dev_out; + fn_pipe_add pipe_add; + fn_pipe_alloc pipe_alloc; + fn_pipe_copy2d pipe_copy2d; + fn_pipe_download pipe_download; + fn_pipe_free pipe_free; + fn_pipe_gemm pipe_gemm; + fn_pipe_peer_copy pipe_peer_copy; + fn_pipe_rmsnorm pipe_rmsnorm; + fn_pipe_rmsnorm_s pipe_rmsnorm_s; + fn_pipe_rope pipe_rope; + fn_pipe_rope_base pipe_rope_base; + fn_pipe_rows_add pipe_rows_add; + fn_pipe_scratch pipe_scratch; + fn_pipe_silu_mul pipe_silu_mul; + fn_pipe_sync pipe_sync; + fn_pipe_upload pipe_upload; + fn_shared_mlp_w4a16 shared_mlp_w4a16; + fn_tensor_update tensor_update; } g_cuda; /* Resolve the DLL and all 11 symbols. Returns 1 on success, 0 otherwise. @@ -82,9 +137,30 @@ static int coli_cuda_load(void){ if(g_cuda.loaded) return g_cuda.available; g_cuda.loaded = 1; - /* Search the model directory first (so a DLL shipped next to the model - * wins), then the engine directory, then the default DLL search path. */ - g_cuda.dll = LoadLibraryA("coli_cuda.dll"); + /* Load coli_cuda.dll from the engine's OWN directory, by absolute path — + * never a bare name. LoadLibraryA("coli_cuda.dll") searches the current + * working directory (and, without SafeDllSearchMode, other writable dirs): + * an attacker who drops a coli_cuda.dll where the user launches glm.exe (or + * inside a downloaded model directory the user cd's into) would get their + * DllMain run at load — classic DLL hijacking -> arbitrary code execution. + * Resolving the path next to glm.exe and loading THAT specific file with + * LOAD_WITH_ALTERED_SEARCH_PATH anchors both the DLL and its dependency + * search to the trusted install directory instead of the CWD. */ + char dllpath[MAX_PATH]; + DWORD mn = GetModuleFileNameA(NULL, dllpath, (DWORD)sizeof(dllpath)); + if(mn > 0 && mn < sizeof(dllpath)){ + char *slash = strrchr(dllpath, '\\'); + if(slash && (size_t)(slash + 1 - dllpath) + sizeof("coli_cuda.dll") <= sizeof(dllpath)){ + strcpy(slash + 1, "coli_cuda.dll"); + g_cuda.dll = LoadLibraryExA(dllpath, NULL, LOAD_WITH_ALTERED_SEARCH_PATH); + } + } + if(!g_cuda.dll){ + /* fallback (GetModuleFileNameA praticamente non fallisce): cerca solo + * nella dir dell'applicazione e in System32, MAI la CWD. */ + g_cuda.dll = LoadLibraryExA("coli_cuda.dll", NULL, + LOAD_LIBRARY_SEARCH_APPLICATION_DIR | LOAD_LIBRARY_SEARCH_SYSTEM32); + } if(!g_cuda.dll){ fprintf(stderr, "[CUDA] coli_cuda.dll not found; GPU tier disabled " "(CPU path remains active).\n"); @@ -119,6 +195,31 @@ static int coli_cuda_load(void){ RESOLVE(tensor_free, fn_tensor_free) RESOLVE(tensor_bytes, fn_tensor_bytes) RESOLVE(tensor_device, fn_tensor_device) + + RESOLVE(attention_absorb_batch, fn_attention_absorb_batch) + RESOLVE(attention_absorb_batch_dev, fn_attention_absorb_batch_dev) + RESOLVE(attention_absorb_kvdev, fn_attention_absorb_kvdev) + RESOLVE(attention_project_batch, fn_attention_project_batch) + RESOLVE(attention_project_batch_dev, fn_attention_project_batch_dev) + RESOLVE(attention_project_batch_dev_out, fn_attention_project_batch_dev_out) + RESOLVE(pipe_add, fn_pipe_add) + RESOLVE(pipe_alloc, fn_pipe_alloc) + RESOLVE(pipe_copy2d, fn_pipe_copy2d) + RESOLVE(pipe_download, fn_pipe_download) + RESOLVE(pipe_free, fn_pipe_free) + RESOLVE(pipe_gemm, fn_pipe_gemm) + RESOLVE(pipe_peer_copy, fn_pipe_peer_copy) + RESOLVE(pipe_rmsnorm, fn_pipe_rmsnorm) + RESOLVE(pipe_rmsnorm_s, fn_pipe_rmsnorm_s) + RESOLVE(pipe_rope, fn_pipe_rope) + RESOLVE(pipe_rope_base, fn_pipe_rope_base) + RESOLVE(pipe_rows_add, fn_pipe_rows_add) + RESOLVE(pipe_scratch, fn_pipe_scratch) + RESOLVE(pipe_silu_mul, fn_pipe_silu_mul) + RESOLVE(pipe_sync, fn_pipe_sync) + RESOLVE(pipe_upload, fn_pipe_upload) + RESOLVE(shared_mlp_w4a16, fn_shared_mlp_w4a16) + RESOLVE(tensor_update, fn_tensor_update) #undef RESOLVE g_cuda.available = 1; @@ -216,4 +317,129 @@ int coli_cuda_tensor_device(const ColiCudaTensor *tensor){ return g_cuda.tensor_device(tensor); } +/* ---- #111 pipeline wrappers ---- */ + + +/* ---- #111 pipeline wrappers (see header for semantics) ---- */ + +int coli_cuda_attention_absorb_batch(ColiCudaTensor *kv_b,float *ctx,const float *q, const float *latent,const float *rope,int S, int H,int Q,int R,int V,int K,int T, float attention_scale){ + if(!g_cuda.available){ return 0; } + return g_cuda.attention_absorb_batch(kv_b, ctx, q, latent, rope, S, H, Q, R, V, K, T, attention_scale); +} + +int coli_cuda_attention_absorb_batch_dev(ColiCudaTensor *kv_b_shard,float *ctx_dev, const float *q_dev,const float *latent_dev,const float *rope_dev, int S,int H,int Q,int R,int V,int K,int T,float scale){ + if(!g_cuda.available){ return 0; } + return g_cuda.attention_absorb_batch_dev(kv_b_shard, ctx_dev, q_dev, latent_dev, rope_dev, S, H, Q, R, V, K, T, scale); +} + +int coli_cuda_attention_absorb_kvdev(ColiCudaTensor *kv_b,float *ctx,const float *q, const float *latent_dev,const float *rope_dev,int H,int Q,int R,int V,int K,int T, float scale){ + if(!g_cuda.available){ return 0; } + return g_cuda.attention_absorb_kvdev(kv_b, ctx, q, latent_dev, rope_dev, H, Q, R, V, K, T, scale); +} + +int coli_cuda_attention_project_batch(ColiCudaTensor *kv_b,ColiCudaTensor *o_proj, float *out,const float *q,const float *latent, const float *rope,int S,int H,int Q,int R, int V,int K,int T,float attention_scale){ + if(!g_cuda.available){ return 0; } + return g_cuda.attention_project_batch(kv_b, o_proj, out, q, latent, rope, S, H, Q, R, V, K, T, attention_scale); +} + +int coli_cuda_attention_project_batch_dev(ColiCudaTensor *kv_b,ColiCudaTensor *o_proj, float *out,const float *q_dev,const float *latent_dev,const float *rope_dev, int S,int H,int Q,int R,int V,int K,int T,float scale){ + if(!g_cuda.available){ return 0; } + return g_cuda.attention_project_batch_dev(kv_b, o_proj, out, q_dev, latent_dev, rope_dev, S, H, Q, R, V, K, T, scale); +} + +int coli_cuda_attention_project_batch_dev_out(ColiCudaTensor *kv_b,ColiCudaTensor *o_proj, float *out_dev,const float *q_dev,const float *latent_dev,const float *rope_dev, int S,int H,int Q,int R,int V,int K,int T,float scale){ + if(!g_cuda.available){ return 0; } + return g_cuda.attention_project_batch_dev_out(kv_b, o_proj, out_dev, q_dev, latent_dev, rope_dev, S, H, Q, R, V, K, T, scale); +} + +int coli_cuda_pipe_add(int device,float *x_dev,const float *t_dev,size_t n){ + if(!g_cuda.available){ return 0; } + return g_cuda.pipe_add(device, x_dev, t_dev, n); +} + +void * coli_cuda_pipe_alloc(int device,size_t bytes){ + if(!g_cuda.available){ return NULL; } + return g_cuda.pipe_alloc(device, bytes); +} + +int coli_cuda_pipe_copy2d(int device,float *dst,int dpitch,const float *src, int spitch,int width,int height){ + if(!g_cuda.available){ return 0; } + return g_cuda.pipe_copy2d(device, dst, dpitch, src, spitch, width, height); +} + +int coli_cuda_pipe_download(int device,const void *src,void *dst,size_t bytes){ + if(!g_cuda.available){ return 0; } + return g_cuda.pipe_download(device, src, dst, bytes); +} + +void coli_cuda_pipe_free(int device,void *p){ + if(!g_cuda.available){ return; } + g_cuda.pipe_free(device, p); +} + +int coli_cuda_pipe_gemm(ColiCudaTensor *t,float *y_dev,const float *x_dev,int S){ + if(!g_cuda.available){ return 0; } + return g_cuda.pipe_gemm(t, y_dev, x_dev, S); +} + +int coli_cuda_pipe_peer_copy(int dst_dev,float *dst,int src_dev, const float *src,size_t bytes){ + if(!g_cuda.available){ return 0; } + return g_cuda.pipe_peer_copy(dst_dev, dst, src_dev, src, bytes); +} + +int coli_cuda_pipe_rmsnorm(int device,float *y_dev,const float *x_dev, const float *w_dev,int S,int D,float eps){ + if(!g_cuda.available){ return 0; } + return g_cuda.pipe_rmsnorm(device, y_dev, x_dev, w_dev, S, D, eps); +} + +int coli_cuda_pipe_rmsnorm_s(int device,float *y_dev,const float *x_dev, const float *w_dev,int S,int D,float eps, int xstride,int ystride){ + if(!g_cuda.available){ return 0; } + return g_cuda.pipe_rmsnorm_s(device, y_dev, x_dev, w_dev, S, D, eps, xstride, ystride); +} + +int coli_cuda_pipe_rope(int device,float *v_dev,const int *pos_dev,int rows, int stride,int offset,int R,int heads,float theta){ + if(!g_cuda.available){ return 0; } + return g_cuda.pipe_rope(device, v_dev, pos_dev, rows, stride, offset, R, heads, theta); +} + +int coli_cuda_pipe_rope_base(int device,float *v_dev,int pos_base,int rows, int stride,int offset,int R,int heads,float theta){ + if(!g_cuda.available){ return 0; } + return g_cuda.pipe_rope_base(device, v_dev, pos_base, rows, stride, offset, R, heads, theta); +} + +int coli_cuda_pipe_rows_add(int device,float *x_dev,const float *partial_dev, const int *rows_dev,int nrows,int D){ + if(!g_cuda.available){ return 0; } + return g_cuda.pipe_rows_add(device, x_dev, partial_dev, rows_dev, nrows, D); +} + +float * coli_cuda_pipe_scratch(int device,int slot,size_t bytes){ + if(!g_cuda.available){ return NULL; } + return g_cuda.pipe_scratch(device, slot, bytes); +} + +int coli_cuda_pipe_silu_mul(int device,float *gate_dev,const float *up_dev,size_t n){ + if(!g_cuda.available){ return 0; } + return g_cuda.pipe_silu_mul(device, gate_dev, up_dev, n); +} + +int coli_cuda_pipe_sync(int device){ + if(!g_cuda.available){ return 0; } + return g_cuda.pipe_sync(device); +} + +int coli_cuda_pipe_upload(int device,void *dst,const void *src,size_t bytes){ + if(!g_cuda.available){ return 0; } + return g_cuda.pipe_upload(device, dst, src, bytes); +} + +int coli_cuda_shared_mlp_w4a16(ColiCudaTensor *gate, ColiCudaTensor *up, ColiCudaTensor *down, float *y, const float *x, int S){ + if(!g_cuda.available){ return 0; } + return g_cuda.shared_mlp_w4a16(gate, up, down, y, x, S); +} + +int coli_cuda_tensor_update(ColiCudaTensor *tensor, const void *weights, const float *scales){ + if(!g_cuda.available){ return 0; } + return g_cuda.tensor_update(tensor, weights, scales); +} + #endif /* _WIN32 */ diff --git a/c/build_cuda.bat b/c/build_cuda.bat new file mode 100644 index 0000000..dc19f09 --- /dev/null +++ b/c/build_cuda.bat @@ -0,0 +1,5 @@ +@echo off +call "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\Build\vcvars64.bat" +cd /d C:\Users\Mark\Desktop\Projects\colibri\c +"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.8\bin\nvcc" -O3 -std=c++17 -arch=sm_120 -Xcompiler=-W3 -shared -DCOLI_CUDA_BUILDING_DLL -L"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.8\lib/x64" -lcudart backend_cuda.cu -o coli_cuda.dll +echo EXITCODE=%ERRORLEVEL% diff --git a/c/coli b/c/coli index d6890ec..4c27415 100755 --- a/c/coli +++ b/c/coli @@ -161,6 +161,34 @@ def resource_request(a, env): def env_for(a): e = dict(os.environ, SNAP=a.model) + if sys.platform == "win32": + # COLI_NO_OMP_TUNE spegne SOLO il blocco OMP (stesso perimetro del + # self-exec di glm.c; presence-based come nel motore: impostarla a + # qualsiasi valore, anche 0, disattiva). I default I/O piu' sotto + # restano attivi: kill-switch dedicati = le var stesse (DIRECT=0 ecc.) + if not e.get("COLI_NO_OMP_TUNE"): + # parita' col tuning OMP self-exec di glm.c (solo Linux/FreeBSD, e + # comunque saltato sotto CUDA/Metal): libgomp legge queste variabili + # prima di main, quindi su Windows vanno nell'ambiente del figlio. + # niente OMP_PROC_BIND/OMP_PLACES: la libgomp di MinGW non supporta + # l'affinity su Windows ("Affinity not supported on this configuration") + from resource_plan import physical_cpu_count + for k, v in (("OMP_WAIT_POLICY", "active"), + ("GOMP_SPINCOUNT", "200000"), + ("OMP_DYNAMIC", "FALSE"), + ("OMP_NUM_THREADS", str(physical_cpu_count()))): + e.setdefault(k, v) + # Default Windows misurati sul box di riferimento (docs/tuning-9950x3d-5090.md), + # tutti lossless e tutti setdefault (un override esplicito vince sempre): + # - DIRECT=1: 10.7 GB/s O_DIRECT vs 9.0 buffered (iobench, anche a cache + # calda); nel motore 0.48 -> 1.02 tok/s. Upstream #162: 1.47x. + # - PIPE=1: overlap load/matmul, +8% sopra DIRECT (byte-identico, riordina + # solo l'I/O). PIPE_WORKERS resta al default 8 (sweep 4/8/16 piatto). + # - PILOT_REAL=1: prefetch cross-layer con load veri (unico prefetch + # funzionante su Windows: fadvise e' no-op), +11%, hit rate +19 punti. + e.setdefault("DIRECT", "1") + e.setdefault("PIPE", "1") + e.setdefault("PILOT_REAL", "1") e["COLI_POLICY"]=a.policy if a.ram: e["RAM_GB"]=str(a.ram) if a.ngen: e["NGEN"]=str(a.ngen) @@ -316,11 +344,50 @@ class Spinner: if self.th: self.th.join(timeout=0.4) if TTY: sys.stdout.write("\r\033[K"); sys.stdout.flush() +def engine_diag(p, errlog=None): + """Lo stdout del motore ha chiuso: il processo e' morto. Dire PERCHE'. + Un SIGKILL dell'OOM-killer del kernel e' altrimenti invisibile — niente errore, + niente exit code, il motore muore muto e sembra un bug nostro (issue #305). + Quasi sempre e' memoria: il picco reale ha sforato la RAM della macchina.""" + try: rc=p.wait(timeout=5) + except Exception: rc=p.poll() + if rc is None: why="its output closed but the process is still alive" + elif rc<0: # POSIX: morte per segnale + try: why=f"killed by {signal.Signals(-rc).name}" + except Exception: why=f"killed by signal {-rc}" + elif rc>0: why=f"exit code {rc}" + else: why="exited cleanly" + print(f"\n {C.yel}[engine terminated: {why}]{C.r}") + if rc is not None and rc<0 and -rc==getattr(signal,"SIGKILL",-1): + print(f" {C.yel}nothing in the engine sends SIGKILL to itself: this is the kernel's\n" + f" OOM-killer. The peak RSS exceeded the machine's free memory.\n" + f" Lower --ram, lower PIN_GB, or shorten the context.{C.r}") + if errlog is not None: + try: + errlog.flush() + for _ in range(20): # il drain thread scrive su errlog quando il child chiude + tail=open(errlog.name).read().strip() + if tail: break + time.sleep(0.05) + if tail: print(f" {C.dgray}" + "\n ".join(tail.splitlines()[-6:]) + f"{C.r}") + except Exception: pass + def stream_turn(p, sentinel, on_bytes): - """legge fino alla sentinella; on_bytes riceve i chunk della risposta. Poi legge la riga STAT.""" - pend=b"" + """legge fino alla sentinella; on_bytes riceve i chunk della risposta. Poi legge la riga STAT. + Il PRIMO Ctrl-C durante lo stream non chiude la sessione: il motore (handler SIGINT) + chiude il turno per la via del tetto NGEN e noi dreniamo fino alla sentinella. + Un SECONDO Ctrl-C esce davvero.""" + pend=b""; interrupted=False while True: - b=p.stdout.read(1) + try: + b=p.stdout.read(1) + except KeyboardInterrupt: + if interrupted or p.poll() is not None: raise + interrupted=True + try: p.send_signal(signal.SIGINT) # non-TTY: il motore potrebbe non aver visto il Ctrl-C + except Exception: pass + print(f"\n {C.yel}⏹ stopping… (Ctrl-C again to quit){C.r}", flush=True) + continue if b==b"": return None pend+=b if pend.endswith(sentinel): @@ -328,7 +395,9 @@ def stream_turn(p, sentinel, on_bytes): if rest: on_bytes(rest) line=p.stdout.readline().decode("utf-8","replace").strip() # STAT tok tps hit rss m=re.match(r"STAT (\S+) (\S+) (\S+) (\S+)", line) - return {"tok":int(m.group(1)),"tps":float(m.group(2)),"hit":float(m.group(3)),"rss":float(m.group(4))} if m else {} + st={"tok":int(m.group(1)),"tps":float(m.group(2)),"hit":float(m.group(3)),"rss":float(m.group(4))} if m else {} + if interrupted: st["interrupted"]=True + return st if len(pend)>len(sentinel): out=pend[:-len(sentinel)]; pend=pend[-len(sentinel):] on_bytes(out) @@ -437,7 +506,11 @@ def cmd_chat(a): if st is None: try: errlog.write(p.stderr.read().decode("utf-8","replace")) except (OSError, ValueError): pass - errlog.seek(0); print(errlog.read()[-1500:]); sys.exit("the engine exited while loading") + errlog.flush(); errlog.seek(0); print(errlog.read()[-1500:]) + engine_diag(p) # perche' e' morto (OOM-kill compreso); errlog e' gia' stampato sopra + sys.exit("the engine exited while loading") + p.stdout.readline() # TIERS line (web-dashboard protocol): emitted once right after STAT; + # left unread it leaks into the first answer's text # READY received. Drain the child's stderr into errlog without blocking: # the engine is still alive (blocked on stdin), so a plain read() would # hang forever waiting for EOF. A short bounded drain grabs the ~400 bytes @@ -464,7 +537,7 @@ def cmd_chat(a): for chunk in textwrap.wrap(l, term_w()-4) or [l]: print(f" {C.dgray}{chunk}{C.r}") except Exception: pass - print(f" {C.dim}type and press Enter · :more continues · :reset clears memory · :q exits{C.r}\n") + print(f" {C.dim}type and press Enter · Ctrl-C stops the answer · :more continues · :reset clears memory · :q exits{C.r}\n") w=term_w()-4 def user_box(msg): """ri-disegna il messaggio dentro una box che si ADATTA su piu' righe: @@ -525,14 +598,17 @@ def cmd_chat(a): st=stream_turn(p, END, echo) if not raw: md.close() sp2.stop() - if st is None: print(f"\n {C.yel}[engine terminated]{C.r}"); break + if st is None: engine_diag(p, errlog); break el=time.time()-t0 if st.get("tok"): print(f"\r {C.dgray}└─ {st['tok']} tok · {st['tps']:.2f} tok/s · hit {st['hit']:.0f}% · RSS {st['rss']:.1f} GB · {el:.0f}s{C.r}") - if st["tok"]>=a.ngen: + if st.get("interrupted"): + print(f" {C.yel}⏹ interrupted; type :more to continue the response{C.r}") + elif st["tok"]>=a.ngen: print(f" {C.yel}…stopped at --ngen ({a.ngen}); type :more to continue the response{C.r}") print() else: + if st.get("interrupted"): print(f" {C.yel}⏹ interrupted{C.r}") print() except KeyboardInterrupt: print(f"\n {C.dim}interrupted{C.r}") @@ -585,7 +661,17 @@ def cmd_bench(a): print(f" {C.dim}downloading missing datasets: {', '.join(missing)}{C.r}") subprocess.call([py, os.path.join(TOOLS,"fetch_benchmarks.py"), "--out", a.data, "--tasks", ",".join(missing), "--limit", str(max(a.limit,200))]) - cmd=[py, os.path.join(TOOLS,"eval_glm.py"), "--snap",a.model, + # il fetch riprova da solo (#304), ma se l'hub resta giu' il bench gira sui task + # disponibili invece di passare a eval file inesistenti. + # EN: the fetch retries on its own (#304), but if the hub stays down the bench + # runs on the available tasks instead of handing eval nonexistent files. + still=[t for t in tasks.split(",") if not os.path.exists(os.path.join(a.data,f"{t}.jsonl"))] + if still: + tasks=",".join(t for t in tasks.split(",") if t not in still) + print(f" {C.yel}skipping (download failed, rerun later): {', '.join(still)}{C.r}") + if not tasks: + print(f" {C.yel}no datasets available — nothing to bench{C.r}"); sys.exit(1) + cmd=[py, os.path.join(TOOLS,"eval_glm.py"), "--glm", GLM, "--snap",a.model, "--tasks", tasks, "--limit", str(a.limit), "--data", a.data] if a.ram: cmd+=["--ram",str(a.ram)] e=env_for(a) @@ -653,7 +739,13 @@ def main(): pw.add_argument(arg,**kw) pw.add_argument("--no-browser",action="store_true",help="don't auto-open the browser") pb=sub.add_parser("bench", parents=[common]); pb.add_argument("tasks", nargs="*") - pb.add_argument("--limit",type=int,default=40); pb.add_argument("--data",default=os.path.join(HERE,"bench")) + pb.add_argument("--limit",type=int,default=40) + if sys.platform == "win32": + _cache_root = os.environ.get("LOCALAPPDATA", os.path.expanduser("~\\AppData\\Local")) + else: + _cache_root = os.environ.get("XDG_CACHE_HOME", os.path.expanduser("~/.cache")) + _bench_cache = os.path.join(_cache_root, "colibri", "bench") + pb.add_argument("--data",default=_bench_cache) pc=sub.add_parser("convert", parents=[common]); pc.add_argument("--repo",default="zai-org/GLM-5.2-FP8") pc.add_argument("--ebits",type=int,default=4); pc.add_argument("--io-bits",type=int,default=8); pc.add_argument("--xbits",type=int,default=0) pc.add_argument("--no-mtp",action="store_true",help="skip the MTP head (no speculative drafts)") diff --git a/c/compat.h b/c/compat.h index 721ba8e..82de8b6 100644 --- a/c/compat.h +++ b/c/compat.h @@ -95,7 +95,18 @@ static inline int compat_open_direct(const char *path){ * prevents 0x0A bytes from being silently translated to \r\n. */ #define COMPAT_O_RDONLY (O_RDONLY | O_BINARY) -/* --- posix_fadvise: no-op (advisory only; safe to ignore) --- */ +/* --- posix_fadvise: Windows has no direct equivalent. Semantics: + * WILLNEED -> warm the OS page cache so a later synchronous pread finds the + * pages resident. Implemented as an overlapped background ReadFile + * into a throwaway scratch buffer (fire-and-forget readahead). Called + * from the dedicated PILOT I/O thread / next-block readahead in moe(), + * NEVER inline on the hot path (the existing comment at glm.c:2847 + * measures inline fadvise submit at ~0.5ms x 169k calls = +92s/48tok). + * Each call owns its OVERLAPPED + scratch buffer -> thread-safe. + * DONTNEED -> no-op: Windows' standby-list trimming self-regulates under pressure, + * and on a low-RAM host keeping the pages is what we want for reuse. + * Matches macOS (compat.h:16-19) which no-ops DONTNEED for the same + * reason. The engine only ever uses DONTNEED as an advisory. */ #ifndef POSIX_FADV_NORMAL #define POSIX_FADV_NORMAL 0 #define POSIX_FADV_RANDOM 1 @@ -104,7 +115,29 @@ static inline int compat_open_direct(const char *path){ #define POSIX_FADV_DONTNEED 4 #define POSIX_FADV_NOREUSE 5 #endif -#define posix_fadvise(fd,off,len,advice) do{(void)(fd);(void)(off);(void)(len);(void)(advice);}while(0) +static inline int compat_fadvise(int fd, off_t off, off_t len, int advice){ + if(advice!=POSIX_FADV_WILLNEED || len<=0) return 0; + intptr_t osfh=_get_osfhandle(fd); + if(osfh==-1 || osfh==-2) return 0; + HANDLE h=(HANDLE)osfh; + /* Cap the readahead window: reading a whole 19MB expert per hint is fine on the + * PILOT thread, but a pathological huge len would spike transient memory. */ + size_t rdlen = (len>(off_t)(64*1024*1024)) ? (size_t)(64*1024*1024) : (size_t)len; + char *buf=(char*)_aligned_malloc(rdlen, 4096); + if(!buf) return -1; + OVERLAPPED ov={0}; + ov.Offset = (DWORD)( (off_t)off & 0xFFFFFFFFULL); + ov.OffsetHigh = (DWORD)(((off_t)off >> 32) & 0xFFFFFFFFULL); + /* Issue an overlapped read. With a non-OVERLAPPED-opened handle ReadFile still + * accepts lpOverlapped (it carries the 64-bit offset) and blocks until the read + * completes — but crucially it populates the standby page cache for this region, + * so the later synchronous pread on the same offsets faults from RAM not disk. */ + DWORD got=0; + ReadFile(h, buf, (DWORD)rdlen, &got, &ov); + _aligned_free(buf); + return 0; +} +#define posix_fadvise compat_fadvise /* --- pread -> ReadFile + OVERLAPPED su raw OS handle --- * Thread-safe (no shared seek position). Gestisce offset >4 GB e chunking diff --git a/c/download_fp8.py b/c/download_fp8.py new file mode 100644 index 0000000..fc8e2b2 --- /dev/null +++ b/c/download_fp8.py @@ -0,0 +1,231 @@ +"""Download GLM-5.2-FP8 from ModelScope (fast, no HF throttling) with +HuggingFace fallback. Parallel shard download, clean progress display. +Usage: python download_fp8.py + python download_fp8.py --parallel 4 + python download_fp8.py --source hf (force HuggingFace) +""" +import os, sys, time, threading, argparse, subprocess + +REPO_MS = "ZhipuAI/GLM-5.2-FP8" # ModelScope +REPO_HF = "zai-org/GLM-5.2-FP8" # HuggingFace +DEST = r"I:\glm52_fp8" + +# ── ANSI colors ── +class C: + dim="\033[2m"; grn="\033[32m"; yel="\033[33m"; cyn="\033[36m"; b="\033[1m"; r="\033[0m" + +def fmt_bytes(n): + if n>=1e9: return f"{n/1e9:.2f} GB" + if n>=1e6: return f"{n/1e6:.1f} MB" + return f"{n/1e3:.0f} KB" + +def fmt_time(s): + if s<60: return f"{s:.0f}s" + if s<3600: return f"{s/60:.0f}m" + return f"{s/3600:.1f}h" + +def bar(cur, total, width=24): + if total<=0: return "["+" "*width+"]" + pct=min(cur/total,1.0); filled=int(width*pct) + return "["+"█"*filled+"░"*(width-filled)+f"] {pct*100:4.0f}%" + +def get_shard_list_hf(): + from huggingface_hub import HfApi + info=HfApi().repo_info(REPO_HF, files_metadata=True) + shards=sorted(s.rfilename for s in info.siblings if s.rfilename.endswith(".safetensors")) + sizes={s.rfilename:s.size for s in info.siblings if s.rfilename.endswith(".safetensors")} + return shards, sizes + +def get_shard_list_ms(): + """Get shard list from ModelScope API.""" + import requests + # ModelScope API: list files + r = requests.get(f"https://modelscope.cn/api/v1/models/{REPO_MS}/repo/files?Revision=master&Root=", timeout=30) + data = r.json()["Data"]["Files"] + shards = sorted(f["Path"] for f in data if f["Path"].endswith(".safetensors")) + sizes = {f["Path"]: f.get("Size", 0) for f in data if f["Path"].endswith(".safetensors")} + return shards, sizes + +def download_file_ms(fn): + """Download a single file from ModelScope using their CDN.""" + from modelscope.hub.file_download import model_file_download + model_file_download( + model_id=REPO_MS, + file_path=fn, + local_dir=DEST, + revision="master", + ) + +def download_file_hf(fn): + """Download a single file from HuggingFace with hf_transfer.""" + os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1" + from huggingface_hub import hf_hub_download + hf_hub_download(REPO_HF, fn, local_dir=DEST) + +def download_file_curl(fn, base_url, expected_size): + """Fallback: download with curl to a .part file with resume.""" + outpath = os.path.join(DEST, fn) + partpath = outpath + ".part" + if os.path.exists(outpath) and os.path.getsize(outpath) == expected_size: + return True + url = f"{base_url}/{fn}" + cmd = ["curl", "-L", "-C", "-", "--retry", "999", "--retry-delay", "5", + "--connect-timeout", "15", "--speed-time", "30", "--speed-limit", "1000", + "-o", partpath, "-H", "User-Agent: colibri-download/1.0", url] + subprocess.run(cmd, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) + if os.path.exists(partpath) and os.path.getsize(partpath) >= expected_size: + os.replace(partpath, outpath) + return True + return False + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--parallel", type=int, default=3) + ap.add_argument("--source", choices=["auto", "ms", "hf"], default="auto", + help="auto (try ModelScope first), ms (ModelScope only), hf (HuggingFace only)") + args = ap.parse_args() + + os.makedirs(DEST, exist_ok=True) + + # Determine source and get shard list + use_ms = False + shards, sizes = [], {} + + if args.source in ("auto", "ms"): + try: + print(f"{C.dim}Trying ModelScope...{C.r}", end=" ", flush=True) + shards, sizes = get_shard_list_ms() + use_ms = True + print(f"{C.grn}✓{C.r} {len(shards)} shards found") + except Exception as e: + print(f"{C.yel}failed ({e}){C.r}") + if args.source == "ms": + print("ModelScope failed and --source ms was set. Exiting."); return + + if not shards: + print(f"{C.dim}Using HuggingFace...{C.r}", end=" ", flush=True) + shards, sizes = get_shard_list_hf() + print(f"{C.grn}✓{C.r} {len(shards)} shards found") + + total = len(shards) + total_bytes = sum(sizes.values()) + source_name = "ModelScope" if use_ms else "HuggingFace" + + # Download metadata files + meta_files = ["config.json", "tokenizer.json", "tokenizer_config.json", + "generation_config.json", "model.safetensors.index.json"] + for fn in meta_files: + out = os.path.join(DEST, fn) + if not os.path.exists(out): + try: + if use_ms: download_file_ms(fn) + else: download_file_hf(fn) + except Exception: pass + + # Build work queue + todo = [] + done_set = set() + for fn in shards: + outpath = os.path.join(DEST, fn) + if os.path.exists(outpath) and os.path.getsize(outpath) == sizes.get(fn, 0): + done_set.add(fn) + else: + todo.append(fn) + + existing = len(done_set) + print(f"\n{C.b}GLM-5.2-FP8 Download ({source_name}){C.r}") + print(f" {C.dim}{total} shards · {total_bytes/1e9:.0f} GB · {existing}/{total} complete{C.r}") + print(f" {C.dim}{len(todo)} to download · {args.parallel} parallel{C.r}") + if todo: + remaining = sum(sizes[fn] for fn in todo) + print(f" {C.dim}Remaining: {remaining/1e9:.0f} GB{C.r}") + print() + + if not todo: + print(f"{C.grn}✓ All shards already downloaded!{C.r}\n"); return + + lock = threading.Lock() + completed = list(done_set) + t0 = time.time() + qidx = [0] + + def worker(wid): + while True: + with lock: + if qidx[0] >= len(todo): return + idx = qidx[0]; qidx[0] += 1 + fn = todo[idx] + + expected = sizes.get(fn, 0) + shard_num = existing + idx + 1 + print(f" {C.cyn}[{shard_num}/{total}]{C.r} {C.dim}{fn}{C.r}") + + success = False + for attempt in range(3): + try: + if use_ms: + download_file_ms(fn) + else: + download_file_hf(fn) + outpath = os.path.join(DEST, fn) + # ModelScope downloads to a cache dir, need to check + # if the file exists at our expected path + if not os.path.exists(outpath): + # Try to find it in ModelScope's cache structure + # and copy/symlink it + pass + if os.path.exists(outpath): + actual = os.path.getsize(outpath) + if expected == 0 or actual == expected: + success = True; break + # Size mismatch — might be in cache + # If we got here, file downloaded but not at expected path + # ModelScope puts it in local_dir; check again + if os.path.exists(outpath): + success = True; break + # Retry with curl fallback + if use_ms: + base = f"https://modelscope.cn/api/v1/models/{REPO_MS}/repo?Revision=master&FilePath=" + else: + base = f"https://huggingface.co/{REPO_HF}/resolve/main" + if download_file_curl(fn, base, expected): + success = True; break + except Exception as e: + if attempt < 2: + print(f" {C.yel}retry {attempt+1}: {e}{C.r}") + time.sleep(3) + else: + print(f" {C.yel}✗ failed: {e}{C.r}") + + with lock: + if success: + completed.append(fn) + elapsed = time.time() - t0 + have = sum(sizes.get(f,0) for f in completed) + pct = 100.0 * have / total_bytes + speed = (have - sum(sizes[f] for f in done_set)) / max(elapsed, 1) + eta = (total_bytes - have) / speed if speed > 0 else 0 + print(f" {C.grn}✓{C.r} {fn} {C.dim}— {len(completed)}/{total} · " + f"{pct:.1f}% · {fmt_time(elapsed)} · ETA {fmt_time(eta)}{C.r}") + else: + print(f" {C.yel}✗ GIVE UP: {fn}{C.r}") + + threads = [threading.Thread(target=worker, args=(i,), daemon=True) for i in range(args.parallel)] + for t in threads: t.start() + for t in threads: t.join() + + print() + final = sum(1 for fn in shards + if os.path.exists(os.path.join(DEST, fn)) + and os.path.getsize(os.path.join(DEST, fn)) == sizes.get(fn, 0)) + if final == total: + print(f"{C.grn}{'='*50}") + print(f" ✓ All {total} shards downloaded!{C.r}\n") + else: + print(f"{C.yel} {final}/{total} complete, {total-final} remaining{C.r}") + print(f" Re-run to resume.\n") + +if __name__ == "__main__": + try: main() + except KeyboardInterrupt: + print(f"\n\n{C.yel}Interrupted. Re-run to resume — no data lost.{C.r}\n") diff --git a/c/glm.c b/c/glm.c index 07fae16..00c7076 100644 --- a/c/glm.c +++ b/c/glm.c @@ -20,6 +20,7 @@ #include #include #include +#include #include #include #include @@ -27,18 +28,26 @@ #include /* PIPE ready-flags/job queue + PILOT_REAL cross-layer handshake */ #include /* sched_yield: PIPE spin / PILOT barrier */ #include -#if defined(__APPLE__) || defined(__linux__) +#if defined(__APPLE__) || defined(__linux__) || defined(__FreeBSD__) #include /* select() serve-loop polling (#68); not on native MinGW */ #endif -#if defined(__APPLE__) || defined(__linux__) +#if defined(__APPLE__) || defined(__linux__) || defined(__FreeBSD__) #include #include /* mlock: inchioda le pagine in RAM / wire pages into RAM */ #include /* fstat per mmap degli shard (COLI_MMAP) */ +#include /* SIGINT = stop morbido del turno in serve mode */ +#endif +#if defined(_WIN32) && (defined(__x86_64__) || defined(__i386__)) +#include /* hwinfo_emit: CPU brand string senza /proc */ #endif #include "st.h" +#ifdef __linux__ +#include "uring.h" +#endif #include "tok.h" #include "tier.h" #include "grammar.h" /* metodo F: draft grammaticali (#48) */ +#include "schema_gbnf.h" /* SCHEMA=: JSON-Schema -> GBNF for method F */ #include "decode_batch.h" #ifdef _OPENMP #include /* scratch per-thread nell'attention */ @@ -94,7 +103,7 @@ typedef struct { * INT4 e' cio' che fa stare la densa residente nei 15 GB (0.5 byte/param). */ /* fmt: 0 F32, 1 INT8, 2 INT4 (2/byte), 3 INT2 (4/byte). q4 ospita sia int4 che int2 packed. */ typedef struct { - int fmt; float *qf; int8_t *q8; uint8_t *q4; float *s; int O, I; + int fmt; float *qf; int8_t *q8; uint8_t *q4; float *s; int O, I, gs; /* gs=group size (0=per-row, 128=grouped) */ #ifdef COLI_CUDA ColiCudaTensor *cuda; #endif @@ -105,13 +114,21 @@ static int64_t qt_bytes(const QT *t){ /* byte residenti del tensore */ if(t->fmt==0) return n*4; if(t->fmt==1) return n + (int64_t)t->O*4; if(t->fmt==3) return (int64_t)t->O*((t->I+3)/4) + (int64_t)t->O*4; - return (int64_t)t->O*((t->I+1)/2) + (int64_t)t->O*4; + if(t->fmt==4){ /* int4 grouped: packed nibbles + O*ceil(I/gs) scales */ + int ng=(t->I+t->gs-1)/t->gs; + return (int64_t)t->O*((t->I+1)/2) + (int64_t)t->O*ng*4; } + return (int64_t)t->O*((t->I+1)/2) + (int64_t)t->O*4; /* fmt=2 int4 per-row */ } typedef struct { float *in_ln, *post_ln; /* MLA (densa, quantizzata) */ QT q_a, q_b, kv_a, kv_b, o; float *q_a_ln, *kv_a_ln; +#ifdef COLI_CUDA + ColiCudaTensor *kv_b_shard[COLI_CUDA_MAX_DEVICES]; + int shard_h0[COLI_CUDA_MAX_DEVICES],shard_hn[COLI_CUDA_MAX_DEVICES],n_kv_b_shard; + int shared_w4a16_failed; +#endif int sparse; /* dense mlp (sparse==0) */ QT gate_proj, up_proj, down_proj; @@ -132,6 +149,9 @@ typedef struct { int *kv_start, max_t; int disk_nrec; char disk_path[2048]; + FILE *disk_fp; /* kept-open handle: fopen once, fwrite per turn, fclose at exit (#4) */ + uint8_t *disk_buf; /* staging buffer: one contiguous record per position (#1) */ + int64_t disk_buf_cap; } KVState; typedef struct { @@ -151,6 +171,7 @@ typedef struct { int *kv_start; /* prima pos valida nella KV del layer (MTP: parziale) */ KVState *kv; ESlot **ecache; int *ecn; int ecap; /* LRU expert per-layer */ + float **kv_dev_L, **kv_dev_R; int *kv_dev_valid; /* ombra KV su device (decode) */ ESlot ws[64]; /* working set del layer corrente (load paralleli) */ ESlot **pin; int *npin; /* HOT-STORE: expert pinnati in RAM (mai evicted) */ uint32_t **eusage; /* contatori persistenti (per STATS/PIN) */ @@ -171,20 +192,32 @@ typedef struct { uint64_t eclock, hits, miss, ereq; uint64_t gpu_expert_calls; int gpu_expert_count; int64_t gpu_expert_bytes; uint64_t n_fw, n_emit; /* metodo E: forward di decode / token emessi */ + uint64_t route_slots, route_swaps; /* CACHE_ROUTE: slots chosen / substituted vs true top-K */ + uint64_t route_agree_hit, route_agree_tot; /* ROUTE_AGREE: |chosen ∩ true top-K| / K */ + double route_kl_sum; uint64_t route_kl_n; /* mean KL(true||chosen) on gate mass */ double t_edisk, t_ewait, t_emm, t_attn, t_kvb, t_head;/* profiling: dove va il tempo */ double t_aproj,t_acore,t_aout; /* attention breakdown */ int64_t resident_bytes; + /* DISK_SPLIT=1: split dei DISK LOAD (miss LRU -> expert_load) per contesto e per tipo + * di layer. ld_ctx: 0=main/verify/prefill, 1=dentro mtp_draft, 2=dentro mtp_absorb. */ + int ld_ctx; + uint64_t miss_draft, miss_absorb; /* miss in moe() per contesto */ + uint64_t ld_mtp, ld_main; /* expert_load per tipo layer (MTP int8 vs main int4) */ + uint64_t bytes_mtp, bytes_main; /* byte letti da disco per tipo layer */ } Model; -static void usage_save(Model *m); +static void usage_save(Model *m); /* cache che impara: definita accanto a stats_dump */ static void tiers_emit(Model *m); static void ehit_mark(Model *m, int layer, int eid); static void emap_emit(Model *m); static void hits_emit(Model *m); -static void hwinfo_emit(Model *m); /* cache che impara: definita accanto a stats_dump */ +static void hwinfo_emit(Model *m); +static int g_repin; +static uint64_t g_last_repin; #ifdef COLI_CUDA static int g_cuda_enabled; static double g_cuda_expert_gb; +static int g_cuda_expert_auto; static int g_cuda_dense; static int g_cuda_release_host; static int g_cuda_devices[COLI_CUDA_MAX_DEVICES], g_cuda_ndev, g_cuda_rr; @@ -198,6 +231,11 @@ static int qt_cuda_upload(QT *t){ : t->fmt==1 ? (const void*)t->q8 : (const void*)t->q4; return coli_cuda_tensor_upload(&t->cuda,weights,t->s,t->fmt,t->I,t->O,t->cuda_device); } +static int qt_cuda_update(QT *t){ + const void *weights=t->fmt==0?(const void*)t->qf: + t->fmt==1?(const void*)t->q8:(const void*)t->q4; + return coli_cuda_tensor_update(t->cuda,weights,t->s); +} static void cuda_stats_print(void){ size_t n=0,b=0; coli_cuda_stats(-1,&n,&b); fprintf(stderr,"[CUDA] resident set: %zu tensors, %.2f GB VRAM\n",n,b/1e9); @@ -355,6 +393,46 @@ static void matmul_i4(float *y, const float *x, const uint8_t *q4, const float * if(i>1]; int lo=(int)(byte&0xF)-8; a += xs[i]*(float)lo; } y[(int64_t)s*O+o]=a*sc; } } } +/* y[S,O] = x[S,I] @ W^T with W int4 packed (2/byte) + per-GROUP scales (fmt=4). + * Same nibble math as matmul_i4, but the scale changes every `gs` elements along I. + * The accumulator resets at each group boundary: dot(x[grp], w[grp]) * scale[grp]. + * gs MUST be a multiple of 16 (the AVX2 vector width). */ +static void matmul_i4_grouped(float *y, const float *x, const uint8_t *q4, const float *scale, + int S, int I, int O, int gs){ + int rb=(I+1)/2; int ng=(I+gs-1)/gs; + #pragma omp parallel for schedule(static) + for(int o=0;oI) glen=I-base; + float sc=scl[g]; + int i=base; +#ifdef __AVX2__ + const __m128i m4=_mm_set1_epi8(0x0F); const __m256i b8=_mm256_set1_epi32(8); + __m256 acc=_mm256_setzero_ps(); + for(; i+16<=base+glen; i+=16){ __m128i by=_mm_loadl_epi64((const __m128i*)(w+(i>>1))); + __m128i lo=_mm_and_si128(by,m4),hi=_mm_and_si128(_mm_srli_epi16(by,4),m4); + __m128i nib=_mm_unpacklo_epi8(lo,hi); + __m256 w0=_mm256_cvtepi32_ps(_mm256_sub_epi32(_mm256_cvtepu8_epi32(nib),b8)); + __m256 w1=_mm256_cvtepi32_ps(_mm256_sub_epi32(_mm256_cvtepu8_epi32(_mm_srli_si128(nib,8)),b8)); + acc=_mm256_fmadd_ps(_mm256_loadu_ps(xs+i), w0, acc); + acc=_mm256_fmadd_ps(_mm256_loadu_ps(xs+i+8), w1, acc); } + a+=hsum256(acc)*sc; +#endif + /* scalar tail for the group remainder */ + for(; i>1]; + a+=(xs[i]*(float)((int)(byte&0xF)-8)+xs[i+1]*(float)((int)(byte>>4)-8))*sc; } + else { uint8_t byte=w[i>>1]; a+=xs[i]*(float)((int)(byte&0xF)-8)*sc; } + } + } + y[(int64_t)s*O+o]=a; + } + } +} /* Decode hot path for gate+up: same exact q4 dot products as matmul_i4, but one * OpenMP dispatch covers both matrices. KTransformers uses persistent pools; * this keeps colibri dependency-free while removing one team launch/expert. */ @@ -366,6 +444,10 @@ static void matmul_i4_pair(float *yg, float *yu, const float *x, for(int z=0;z<2*O;z++){ int o=z>1]; a+=x[i]*(float)((b&15)-8)+x[i+1]*(float)((b>>4)-8); } if(i>1]&15)-8); @@ -401,8 +486,24 @@ static int g_no_fused_pair=0; /* COLI_NO_FUSED_PAIR=1: disable the gate+up kern * that changes OMP scheduling vs separate matmul_qt calls — this * shifts floating-point accumulation order and can collapse MTP * draft acceptance by flipping near-ties (#163). */ +/* #163: l'acceptance MTP crolla quando il forward di draft (S=1) e quello di verifica + * (S>=2) non calcolano la STESSA funzione. Tre interruttori dipendono da S: il gate + * int4-IDOT (S>=g_i4s — asimmetrico proprio dove g_i4s>1), la fusione gate+up solo-S==1, + * e la soglia righe del GEMM Metal. Con SPEC_PIN=1 (default) ogni forward emesso mentre + * i draft del modello sono attivi resta sulla famiglia di kernel di S=1: draft e verifica + * coincidono per costruzione. Prefill e decode non speculativo sono intoccati. + * EN: MTP acceptance collapses when the draft (S=1) and verify (S>=2) forwards do not + * compute the SAME function. Three switches are S-dependent: the int4 IDOT gate + * (S>=g_i4s — asymmetric exactly on ISAs where g_i4s>1), the S==1-only gate+up fusion, + * and the Metal GEMM row threshold. SPEC_PIN=1 (default) pins every forward issued + * while model drafts are live to the platform's S=1 kernel family, so draft and verify + * agree by construction; prefill and non-speculative decode are untouched. + * SPEC_PIN=0 restores the S-dependent gates (A/B). */ +static int g_spec_pin=1; +static int g_spec_live=0; /* set by spec_decode while drafts are live */ +static inline int spec_pinned(void){ return g_spec_pin && g_spec_live; } static void expert_gate_up(float *g,float *u,const float *x,QT *wg,QT *wu,int S){ - if(!g_no_fused_pair&&S==1&&wg->fmt==2&&wu->fmt==2&&wg->I==wu->I&&wg->O==wu->O) + if(!g_no_fused_pair&&!spec_pinned()&&S==1&&wg->fmt==2&&wu->fmt==2&&wg->I==wu->I&&wg->O==wu->O) matmul_i4_pair(g,u,x,wg->q4,wg->s,wu->q4,wu->s,wg->I,wg->O); else { matmul_qt(g,x,wg,S); matmul_qt(u,x,wu,S); } } @@ -454,6 +555,8 @@ static void matmul_i2(float *y, const float *x, const uint8_t *q2, const float * #define IDOT_KERNEL "avx-vnni" #elif defined(__AVX2__) #define IDOT_KERNEL "avx2" +#elif defined(__ARM_NEON) && defined(__ARM_FEATURE_MATMUL_INT8) +#define IDOT_KERNEL "neon-i8mm" #elif defined(__ARM_NEON) #define IDOT_KERNEL "neon" #elif defined(__VSX__) @@ -536,18 +639,30 @@ static inline int32_t dot_i8i8(const int8_t *w, const int8_t *x, int I){ #elif defined(__ARM_NEON) /* ARM: SDOT nativo se disponibile (Apple Silicon: sempre); altrimenti vmull/vpadal. * Stesso bound anti-overflow del trucco AVX2: coppie <= 128*127*2 = 32512 < 32767. */ +#if defined(__ARM_FEATURE_DOTPROD) + /* 4 accumulatori indipendenti: SDOT ha latenza ~3-4 cicli, con un solo acc la + * catena seriale strozza il core a ~26 GB/s di pesi; con 4 lane indipendenti il + * dot diventa memory-bound (misurato su M4: 26 -> 63 GB/s per core, 2.4x). */ + int32x4_t a0=vdupq_n_s32(0),a1=vdupq_n_s32(0),a2=vdupq_n_s32(0),a3=vdupq_n_s32(0); + for(;i+64<=I;i+=64){ + a0=vdotq_s32(a0,vld1q_s8(w+i), vld1q_s8(x+i)); + a1=vdotq_s32(a1,vld1q_s8(w+i+16),vld1q_s8(x+i+16)); + a2=vdotq_s32(a2,vld1q_s8(w+i+32),vld1q_s8(x+i+32)); + a3=vdotq_s32(a3,vld1q_s8(w+i+48),vld1q_s8(x+i+48)); + } + int32x4_t acc=vaddq_s32(vaddq_s32(a0,a1),vaddq_s32(a2,a3)); + for(;i+16<=I;i+=16) acc=vdotq_s32(acc,vld1q_s8(w+i),vld1q_s8(x+i)); + sum=vaddvq_s32(acc); +#else int32x4_t acc=vdupq_n_s32(0); for(;i+16<=I;i+=16){ int8x16_t wv=vld1q_s8(w+i), xv=vld1q_s8(x+i); -#if defined(__ARM_FEATURE_DOTPROD) - acc=vdotq_s32(acc,wv,xv); -#else int16x8_t p=vmull_s8(vget_low_s8(wv),vget_low_s8(xv)); p=vmlal_s8(p,vget_high_s8(wv),vget_high_s8(xv)); acc=vpadalq_s16(acc,p); -#endif } sum=vaddvq_s32(acc); +#endif #elif defined(__VSX__) /* POWER8: vec_msum (s8 x u8 -> s32) somma i prodotti byte DIRETTAMENTE in lane * s32, 16 byte/iter: il bound anti-saturazione a 16 bit di maddubs qui non serve. @@ -627,6 +742,28 @@ static inline int32_t dot_i4i8(const uint8_t *w4, const int8_t *x, int I){ sum=hsum256_i32(acc); #elif defined(__ARM_NEON) const uint8x16_t m4q=vdupq_n_u8(0x0F); const int8x16_t b8q=vdupq_n_s8(8); +#if defined(__ARM_FEATURE_DOTPROD) + /* 4 accumulatori indipendenti (vedi dot_i8i8): spezza la catena seriale su acc. + * Misurato su M4: 12.4 -> 29.9 GB/s di pesi per core (2.4x). */ + int32x4_t a0=vdupq_n_s32(0),a1=vdupq_n_s32(0),a2=vdupq_n_s32(0),a3=vdupq_n_s32(0); + for(;i+64<=I;i+=64){ + uint8x16_t byA=vld1q_u8(w4+(i>>1)), byB=vld1q_u8(w4+(i>>1)+16); + uint8x16x2_t zA=vzipq_u8(vandq_u8(byA,m4q), vshrq_n_u8(byA,4)); /* nibble in ordine */ + uint8x16x2_t zB=vzipq_u8(vandq_u8(byB,m4q), vshrq_n_u8(byB,4)); + a0=vdotq_s32(a0,vsubq_s8(vreinterpretq_s8_u8(zA.val[0]),b8q),vld1q_s8(x+i)); + a1=vdotq_s32(a1,vsubq_s8(vreinterpretq_s8_u8(zA.val[1]),b8q),vld1q_s8(x+i+16)); + a2=vdotq_s32(a2,vsubq_s8(vreinterpretq_s8_u8(zB.val[0]),b8q),vld1q_s8(x+i+32)); + a3=vdotq_s32(a3,vsubq_s8(vreinterpretq_s8_u8(zB.val[1]),b8q),vld1q_s8(x+i+48)); + } + int32x4_t acc=vaddq_s32(vaddq_s32(a0,a1),vaddq_s32(a2,a3)); + for(;i+32<=I;i+=32){ + uint8x16_t by=vld1q_u8(w4+(i>>1)); /* 16 byte = 32 nibble */ + uint8x16x2_t z=vzipq_u8(vandq_u8(by,m4q), vshrq_n_u8(by,4)); /* nibble in ordine */ + acc=vdotq_s32(acc,vsubq_s8(vreinterpretq_s8_u8(z.val[0]),b8q),vld1q_s8(x+i)); + acc=vdotq_s32(acc,vsubq_s8(vreinterpretq_s8_u8(z.val[1]),b8q),vld1q_s8(x+i+16)); + } + sum=vaddvq_s32(acc); +#else int32x4_t acc=vdupq_n_s32(0); for(;i+32<=I;i+=32){ uint8x16_t by=vld1q_u8(w4+(i>>1)); /* 16 byte = 32 nibble */ @@ -634,18 +771,15 @@ static inline int32_t dot_i4i8(const uint8_t *w4, const int8_t *x, int I){ int8x16_t w0=vsubq_s8(vreinterpretq_s8_u8(z.val[0]),b8q); int8x16_t w1=vsubq_s8(vreinterpretq_s8_u8(z.val[1]),b8q); int8x16_t x0=vld1q_s8(x+i), x1=vld1q_s8(x+i+16); -#if defined(__ARM_FEATURE_DOTPROD) - acc=vdotq_s32(acc,w0,x0); acc=vdotq_s32(acc,w1,x1); -#else int16x8_t p=vmull_s8(vget_low_s8(w0),vget_low_s8(x0)); /* |w|<=8: nessun overflow */ p=vmlal_s8(p,vget_high_s8(w0),vget_high_s8(x0)); acc=vpadalq_s16(acc,p); p=vmull_s8(vget_low_s8(w1),vget_low_s8(x1)); p=vmlal_s8(p,vget_high_s8(w1),vget_high_s8(x1)); acc=vpadalq_s16(acc,p); -#endif } sum=vaddvq_s32(acc); +#endif #elif defined(__VSX__) /* 16 byte = 32 nibble. vec_mergeh/vec_mergel su ppc64le (GCC) interallacciano come * unpacklo/unpackhi x86 (verificato empiricamente su POWER8): i nibble escono in @@ -676,8 +810,131 @@ static inline int32_t dot_i4i8(const uint8_t *w4, const int8_t *x, int I){ if(i>1]; sum+=((int)(b&0xF)-8)*x[i]; } return sum; } +#if defined(__ARM_NEON) && defined(__ARM_FEATURE_MATMUL_INT8) +/* SMMLA (i8mm): vmmlaq_s32 vede ogni int8x16_t come matrice 2x8 row-major (byte 0-7 = + * riga 0, byte 8-15 = riga 1) e accumula C += A*B^T nel 2x2 int32: lane0=a0.b0, + * lane1=a0.b1, lane2=a1.b0, lane3=a1.b1. vcombine di due mezze-righe costruisce la + * matrice: A = due righe di peso (o,o+1), B = due righe di attivazione (s,s+1), quindi + * meta' traffico pesi e doppio lavoro per istruzione a S>=2. EN: vmmlaq_s32 treats each + * int8x16_t as a 2x8 row-major matrix and does C += A*B^T on a 2x2 int32 tile; vcombine + * of vget_low/high halves builds the 2-row register from two weight/activation rows. */ +static inline int32x4_t mm_tile16(int32x4_t acc, int8x16_t wo, int8x16_t wo1, + int8x16_t xs, int8x16_t xs1){ + acc=vmmlaq_s32(acc, vcombine_s8(vget_low_s8(wo), vget_low_s8(wo1)), + vcombine_s8(vget_low_s8(xs), vget_low_s8(xs1))); + return vmmlaq_s32(acc, vcombine_s8(vget_high_s8(wo), vget_high_s8(wo1)), + vcombine_s8(vget_high_s8(xs), vget_high_s8(xs1))); +} +static void matmul_q_idot_mm(float *y, const int8_t *xq, const float *sx, const int8_t *q, + const float *scale, int S, int I, int O){ + #pragma omp parallel for schedule(static) + for(int o=0;o<(O&~1);o+=2){ + const int8_t *wo=q+(int64_t)o*I, *wo1=q+(int64_t)(o+1)*I; + float sc0=scale[o], sc1=scale[o+1]; + for(int s=0;s<(S&~1);s+=2){ + const int8_t *xs=xq+(int64_t)s*I, *xs1=xq+(int64_t)(s+1)*I; + /* 4 accumulatori indipendenti: una sola catena vmmla e' latency-bound. + * EN: 4 independent accumulators; a single vmmla chain is latency-bound. */ + int32x4_t a0=vdupq_n_s32(0),a1=vdupq_n_s32(0),a2=vdupq_n_s32(0),a3=vdupq_n_s32(0); int i=0; + for(;i+64<=I;i+=64){ + a0=mm_tile16(a0,vld1q_s8(wo+i), vld1q_s8(wo1+i), vld1q_s8(xs+i), vld1q_s8(xs1+i)); + a1=mm_tile16(a1,vld1q_s8(wo+i+16),vld1q_s8(wo1+i+16),vld1q_s8(xs+i+16),vld1q_s8(xs1+i+16)); + a2=mm_tile16(a2,vld1q_s8(wo+i+32),vld1q_s8(wo1+i+32),vld1q_s8(xs+i+32),vld1q_s8(xs1+i+32)); + a3=mm_tile16(a3,vld1q_s8(wo+i+48),vld1q_s8(wo1+i+48),vld1q_s8(xs+i+48),vld1q_s8(xs1+i+48)); + } + for(;i+16<=I;i+=16) + a0=mm_tile16(a0,vld1q_s8(wo+i),vld1q_s8(wo1+i),vld1q_s8(xs+i),vld1q_s8(xs1+i)); + int32x4_t acc=vaddq_s32(vaddq_s32(a0,a1),vaddq_s32(a2,a3)); + int32_t d00=vgetq_lane_s32(acc,0), d01=vgetq_lane_s32(acc,1); + int32_t d10=vgetq_lane_s32(acc,2), d11=vgetq_lane_s32(acc,3); + for(;i>1)), byo1=vld1q_u8(wo1+(i>>1)); + uint8x16_t cyo=vld1q_u8(wo+(i>>1)+16), cyo1=vld1q_u8(wo1+(i>>1)+16); + uint8x16x2_t zo =vzipq_u8(vandq_u8(byo, m4q), vshrq_n_u8(byo, 4)); + uint8x16x2_t zo1=vzipq_u8(vandq_u8(byo1,m4q), vshrq_n_u8(byo1,4)); + uint8x16x2_t ko =vzipq_u8(vandq_u8(cyo, m4q), vshrq_n_u8(cyo, 4)); + uint8x16x2_t ko1=vzipq_u8(vandq_u8(cyo1,m4q), vshrq_n_u8(cyo1,4)); + a0=mm_tile16(a0, vsubq_s8(vreinterpretq_s8_u8(zo.val[0]),b8q), + vsubq_s8(vreinterpretq_s8_u8(zo1.val[0]),b8q), + vld1q_s8(xs+i), vld1q_s8(xs1+i)); + a1=mm_tile16(a1, vsubq_s8(vreinterpretq_s8_u8(zo.val[1]),b8q), + vsubq_s8(vreinterpretq_s8_u8(zo1.val[1]),b8q), + vld1q_s8(xs+i+16), vld1q_s8(xs1+i+16)); + a2=mm_tile16(a2, vsubq_s8(vreinterpretq_s8_u8(ko.val[0]),b8q), + vsubq_s8(vreinterpretq_s8_u8(ko1.val[0]),b8q), + vld1q_s8(xs+i+32), vld1q_s8(xs1+i+32)); + a3=mm_tile16(a3, vsubq_s8(vreinterpretq_s8_u8(ko.val[1]),b8q), + vsubq_s8(vreinterpretq_s8_u8(ko1.val[1]),b8q), + vld1q_s8(xs+i+48), vld1q_s8(xs1+i+48)); + } + for(;i+32<=I;i+=32){ + uint8x16_t byo=vld1q_u8(wo+(i>>1)), byo1=vld1q_u8(wo1+(i>>1)); + uint8x16x2_t zo =vzipq_u8(vandq_u8(byo, m4q), vshrq_n_u8(byo, 4)); + uint8x16x2_t zo1=vzipq_u8(vandq_u8(byo1,m4q), vshrq_n_u8(byo1,4)); + a0=mm_tile16(a0, vsubq_s8(vreinterpretq_s8_u8(zo.val[0]),b8q), + vsubq_s8(vreinterpretq_s8_u8(zo1.val[0]),b8q), + vld1q_s8(xs+i), vld1q_s8(xs1+i)); + a1=mm_tile16(a1, vsubq_s8(vreinterpretq_s8_u8(zo.val[1]),b8q), + vsubq_s8(vreinterpretq_s8_u8(zo1.val[1]),b8q), + vld1q_s8(xs+i+16), vld1q_s8(xs1+i+16)); + } + int32x4_t acc=vaddq_s32(vaddq_s32(a0,a1),vaddq_s32(a2,a3)); + int32_t d00=vgetq_lane_s32(acc,0), d01=vgetq_lane_s32(acc,1); + int32_t d10=vgetq_lane_s32(acc,2), d11=vgetq_lane_s32(acc,3); + for(;i+1>1], bo1=wo1[i>>1]; + int a0=(int)(bo&0xF)-8, a1=(int)(bo>>4)-8, b0=(int)(bo1&0xF)-8, b1=(int)(bo1>>4)-8; + int u0=xs[i],u1=xs[i+1],v0=xs1[i],v1=xs1[i+1]; + d00+=a0*u0+a1*u1; d01+=a0*v0+a1*v1; d10+=b0*u0+b1*u1; d11+=b0*v0+b1*v1; } + if(i>1], bo1=wo1[i>>1]; + int a0=(int)(bo&0xF)-8, b0=(int)(bo1&0xF)-8; + d00+=a0*xs[i]; d01+=a0*xs1[i]; d10+=b0*xs[i]; d11+=b0*xs1[i]; } + y[(int64_t)s*O+o] =(float)d00*sc0*sx[s]; + y[(int64_t)s*O+(o+1)] =(float)d10*sc1*sx[s]; + y[(int64_t)(s+1)*O+o] =(float)d01*sc0*sx[s+1]; + y[(int64_t)(s+1)*O+(o+1)]=(float)d11*sc1*sx[s+1]; + } + if(S&1){ int s=S-1; const int8_t *xs=xq+(int64_t)s*I; + y[(int64_t)s*O+o] =(float)dot_i4i8(wo, xs,I)*sc0*sx[s]; + y[(int64_t)s*O+(o+1)]=(float)dot_i4i8(wo1,xs,I)*sc1*sx[s]; } + } + if(O&1){ int o=O-1; const uint8_t *w=q4+(int64_t)o*rb; float sc=scale[o]; + #pragma omp parallel for schedule(static) + for(int s=0;s=2){ matmul_q_idot_mm(y,xq,sx,q,scale,S,I,O); return; } +#endif #pragma omp parallel for schedule(static) for(int o=0;o=2){ matmul_i4_idot_mm(y,xq,sx,q4,scale,S,I,O); return; } +#endif #pragma omp parallel for schedule(static) for(int o=0;o -5160.47 (IDOT) = +0.117 nat/token, + * ~+12% perplexity. Gli altri matmul del prefill (o_proj, kv_b, expert) tengono l'IDOT. + * EN: allow_idot=0 forces the EXACT int4/int8 kernel (f32 activations). The attention + * projections need it: IDOT's int8 activation quantization costs +0.117 nats/token there + * (~+12% perplexity), measured. Every other prefill matmul keeps IDOT as before. */ +static void matmul_qt_ex(float *y, const float *x, QT *w, int S, int allow_idot); +static void matmul_qt(float *y, const float *x, QT *w, int S){ matmul_qt_ex(y,x,w,S,1); } +static void matmul_qt_ex(float *y, const float *x, QT *w, int S, int allow_idot){ #ifdef COLI_METAL /* Large row-batches (prefill: kv_b reconstruction, o_proj, dense MLP, step_all logits) * amortize Metal's ~5ms submit latency; small-S decode matmuls stay on CPU (NEON wins). * Weights must be registered (all dense QT allocs are, via qalloc). */ - if(g_metal_enabled && S>=g_metal_gemm_min && (w->fmt==1||w->fmt==2) && !omp_in_parallel()){ + if(g_metal_enabled && S>=g_metal_gemm_min && !spec_pinned() && (w->fmt==1||w->fmt==2) && !omp_in_parallel()){ const void *wp = w->fmt==1 ? (const void*)w->q8 : (const void*)w->q4; if(coli_metal_gemm(y,x,wp,w->s,w->fmt,S,w->I,w->O)) return; } @@ -731,13 +1000,22 @@ static void matmul_qt(float *y, const float *x, QT *w, int S){ } #endif if(w->fmt==0){ matmul(y,x,w->qf,S,w->I,w->O); return; } + /* fmt=4: grouped int4 — always use the exact grouped kernel (no IDOT approximation, + * since the whole point of grouped scales is better quality). */ + if(w->fmt==4){ matmul_i4_grouped(y,x,w->q4,w->s,S,w->I,w->O,w->gs); return; } /* int8 IDOT vince sempre (1.4-2.5x). int4 IDOT: l'autore su AVX2 trovo' che a S=1 * non ripaga (soglia S>=2); ma su ARM/SDOT il singolo token CONVIENE (vedi g_i4s / * PR #9 per il gemello VNNI). Soglia configurabile con I4S. * EN: int8 IDOT always wins (1.4-2.5x). int4 IDOT: on AVX2 the author found S=1 didn't * pay (S>=2 gate); on ARM/SDOT single-token DOES pay (see g_i4s / PR #9 for the VNNI * twin). Threshold configurable via I4S. */ - if(g_idot && (w->fmt==1 || (w->fmt==2 && S>=g_i4s))){ + /* #163: sotto SPEC_PIN il gate int4-IDOT usa la decisione di S=1 per OGNI S, cosi' + * draft e verifica restano sulla stessa famiglia. (CUDA non e' toccato: la sua + * condizione non dipende da S, quindi e' gia' coerente tra draft e verifica.) + * EN: under SPEC_PIN the int4 IDOT gate uses the S=1 decision for EVERY S, so draft + * and verify stay in one family. (CUDA untouched: its condition is S-independent, + * hence already draft/verify-consistent.) */ + if(allow_idot && g_idot && (w->fmt==1 || (w->fmt==2 && (spec_pinned() ? g_i4s<=1 : S>=g_i4s)))){ int I=w->I; int8_t *xq; float *sx; if(S<0 || I<0 || (size_t)S>SIZE_MAX/(size_t)(I?I:1)){ fprintf(stderr,"matmul_qt: shape overflow\n"); exit(1); } quant_scratch((size_t)S*I,(size_t)S,&xq,&sx); @@ -809,6 +1087,20 @@ static float g_temp=-1; /* TEMP: temperatura di sampling sui TOKEN. <0 = auto ( static float g_nuc=0.95f;/* NUCLEUS: top-p sul vocabolario (default dal generation_config GLM-5.2) */ static int g_topk=0; /* TOPK=n -> usa n expert/token invece di config (ricerca: meno disco) */ static float g_topp=0; /* TOPP=p (0..1) -> top-p adattivo: tieni gli expert fino a peso cumulato p */ +static int g_expert_budget=0; /* EXPERT_BUDGET=N -> cap distinct experts loaded per layer across the + * batch-union. Reduces disk I/O on cold/low-RAM hosts by dropping the + * lowest-gate-weight experts from the cross-position union. MoE-Spec + * (arXiv 2602.16052): top-32 of 64 capture 93% routing weight. */ +static int64_t g_budget_dropped=0; /* total experts dropped by EXPERT_BUDGET across all layers */ +/* CACHE_ROUTE (paper 2412.00099 max-rank): opt-in only. Keep true top-J always; + * fill remaining slots preferring pin∪LRU experts ranked within top-M (or mass ROUTE_P). */ +static int g_cache_route=0; +static int g_route_j=2; /* ROUTE_J: sacred top ranks (always take, even uncached) */ +static int g_route_m=12; /* ROUTE_M: max-rank window for cache-preferring fill */ +static float g_route_p=0; /* ROUTE_P: if >0, choose M from cumulative router mass instead */ +static float g_route_alpha=1.f; /* ROUTE_ALPHA: scale gate mass of CACHE_ROUTE substitutes before renorm (1=off) */ +static int g_route_agree=0; /* ROUTE_AGREE=1: footer overlap% + mean KL vs true top-K */ +static int expert_is_resident(Model *m, int layer, int eid); /* pin∪LRU; defined near pilot */ static int g_spec=1; /* metodo C: SPEC=0 disabilita il prefetch speculativo cross-layer */ static int g_draft=0; /* metodo E: DRAFT=n token auto-speculati per forward via n-gram lookup * (0=off). LOSSLESS: verifica = output identico al greedy. Default OFF: @@ -856,10 +1148,15 @@ static int g_looka=0; /* LOOKA=1: misura (solo contatori, zero effetti) quant * [2] post-attention del layer L -> routing di L+1 (un residuo MoE e * un'attention di anticipo: il punto dove il prefetch avrebbe * un intero giro di disco per lavorare in ombra). */ -static int64_t la_hit[3], la_tot[3]; -static int la_pred[2][130][16]; static signed char la_val[2][130]; +static int64_t la_hit[4], la_tot[4]; /* [0]=prev, [1]=skip-attn, [2]=PILOT, [3]=two-step */ +static int la_pred[3][130][16]; static signed char la_val[3][130]; static int g_pilot=0; /* PILOT=1: prefetch pilotato dal router (vedi pilot_prefetch) */ static int g_pilot_k=8; /* PILOT_K=k: prefetcha solo le prime k predizioni per posizione */ +static int g_disk_split=0; /* DISK_SPLIT=1: contatori che spezzano i DISK LOAD (miss LRU) in + * draft MTP / absorb / verify-main e in layer MTP (int8) vs main + * (int4), con i byte letti. Default OFF: a flag spento gli atomic + * non vengono MAI toccati (zero overhead), le righe extra di stats + * non vengono stampate. Solo misura: nessun effetto sull'output. */ /* Aligned allocator for dense QT weights/scales: under METAL, page-align + register so the * GPU reads them zero-copy (no upload duplicate). Plain malloc otherwise. */ static void *qalloc(size_t n){ @@ -873,6 +1170,9 @@ static void *qalloc(size_t n){ static float *qsalloc(int O){ return (float*)qalloc((size_t)O*sizeof(float)); } static int g_pilot_real=0;/* PILOT_REAL=1: il pilota fa LOAD VERI cross-layer dentro ecache[L+1] * (non il semplice WILLNEED). Implica PILOT=1. Default OFF: hint-only. */ +static int g_pilot_two=0; /* PILOT_TWO=1: two-step prefetch — before running L+1's router, + * approximate MoE(L) using only the shared expert (resident, no disk) + * and add it to the state. Trades 3 small matmuls for +2.3% recall. */ /* Handshake main<->pilota per il load-vero cross-layer. Invariante di sicurezza in DUE parti: * 1) Percorso MATMUL (moe): il pilota scrive SOLO ecache[layer] con layer > g_cur_moe_layer; * il matmul in moe() legge SOLO ecache[layer]==g_cur_moe_layer, e la barriera a inizio moe() @@ -885,8 +1185,9 @@ static int g_pilot_real=0;/* PILOT_REAL=1: il pilota fa LOAD VERI cross-layer de * niente torn read di ecn[]/eid. Il pilota non altera MAI il valore di un expert, solo QUALE * expert e' residente: con un load andato a buon fine l'output resta byte-identico all'OFF. */ static pthread_mutex_t g_pilot_mx=PTHREAD_MUTEX_INITIALIZER; +static pthread_cond_t g_pilot_cv=PTHREAD_COND_INITIALIZER; static _Atomic int g_cur_moe_layer=-1; /* massimo layer moe in cui il MAIN e' entrato (per forward) */ -static _Atomic int g_pilot_inflight=-1; /* layer che il worker sta REAL-caricando adesso (-1 = idle) */ +static int g_pilot_inflight[256]; /* protected by g_pilot_mx; URING can load a layer concurrently */ static _Atomic long g_pilot_loads=0; /* load cross-layer VERI completati (banda spesa) */ static _Atomic long g_pilot_drops=0; /* predizioni scartate perche' il main possiede gia' il layer */ /* sceglie il formato da `bits`: >=16 f32, 5..8 int8, <=4 int4-packed */ @@ -923,14 +1224,21 @@ static inline float siluf(float x){ return x/(1.f+expf(-x)); } /* RoPE interleaved su un vettore di dimensione qk_rope a posizione pos */ static void rope_interleave(float *v, int pos, const Cfg *c){ int half = c->qk_rope/2; - /* Validate against the fixed buffer: the config checker allows qk_rope up - * to 1<<16 but this stack buffer holds 256 floats. Abort cleanly instead - * of smashing the stack. (GLM-5.2 qk_rope=64, well within bounds.) */ + /* Validate against the fixed buffers (in[256], cache cs/sn[128] -> qk_rope<=256). + * Abort cleanly instead of smashing the stack. (GLM-5.2 qk_rope=64.) (#183) */ if(c->qk_rope > 256){ fprintf(stderr,"qk_rope=%d exceeds rope_interleave buffer (256)\n",c->qk_rope); exit(1); } + typedef struct { int pos,qk,valid; float theta,cs[128],sn[128]; } RopeCache; /* (#80) */ + static _Thread_local RopeCache cache; float in[256]; memcpy(in,v,c->qk_rope*sizeof(float)); + if(!cache.valid||cache.pos!=pos||cache.qk!=c->qk_rope||cache.theta!=c->theta){ + for(int j=0;jtheta,-2.0f*j/c->qk_rope),ang=pos*inv; + cache.cs[j]=cosf(ang); cache.sn[j]=sinf(ang); + } + cache.pos=pos; cache.qk=c->qk_rope; cache.theta=c->theta; cache.valid=1; + } for(int j=0;jtheta, -2.0f*j/c->qk_rope); - float ang = pos*inv, cs=cosf(ang), sn=sinf(ang); + float cs=cache.cs[j],sn=cache.sn[j]; float a=in[2*j], b=in[2*j+1]; v[j] = a*cs - b*sn; v[half+j] = b*cs + a*sn; @@ -999,6 +1307,29 @@ static void load_cfg(Cfg *c, const char *snap){ free(ar); } +/* Derive the fmt=4 group size from the scale-array byte count. A grouped-int4 + * tensor stores ceil(I/gs) f32 scales per output row, so: + * ns_bytes == O * ceil(I/gs) * 4 => gs == I * 4 / (ns_bytes/O - ... ) + * We probe candidate group sizes (must be a multiple of 16, the AVX2 vector + * width the grouped kernel requires) from finest to coarsest and return the + * first whose predicted scale-array size matches ns_bytes. Returns 0 if no + * candidate fits (then it's plain per-row int4, fmt=2, not grouped). + * Data-driven: g64/g128/g256 all just work; adding a size means listing it. */ +static int detect_group_size(int O, int I, int64_t ns){ + if(O<=0 || ns<=(int64_t)O*4 || I<=0) return 0; /* not grouped */ + /* ns/O is the per-row scale bytes; groups = (ns/O)/4; gs = ceil(I/groups). + * Probe from small gs (finest granularity) upward so the most granular + * match wins — that's what we want, since finer groups are unambiguous. */ + static const int cands[]={16,32,48,64,96,128,192,256}; + for(int ci=0; ci<(int)(sizeof(cands)/sizeof(cands[0])); ci++){ + int gs=cands[ci]; + if(gs>I) break; + int ng=(I+gs-1)/gs; + if(ns==(int64_t)O*ng*4) return gs; + } + return 0; +} + /* costruisce un QT [O,I] dal disco in `t` (buffer riusabili tra chiamate). * - se esiste `name.qs`: pesi GIA' quantizzati nel container (U8 qdata + F32 scala) -> letti diretti * - altrimenti: tensore pieno (f32/bf16) -> quantizzato a runtime a `bits` (oracolo tiny / pesi pieni) @@ -1007,9 +1338,18 @@ static void qt_from_disk(Model *m, const char *name, int O, int I, int bits, int char sn[300]; snprintf(sn,sizeof(sn),"%s.qs",name); if(st_has(&m->S,sn)){ int64_t nb=st_nbytes(&m->S,name); - int fmt = (nb==(int64_t)O*I)?1 : (nb==(int64_t)O*((I+1)/2))?2 : 3; /* int8 / int4 / int2 dai byte */ - if(fmt==1){ if(t->fmt!=1||!t->q8){ t->fmt=1; t->O=O; t->I=I; t->q8=qalloc(nb); t->s=qsalloc(O); } st_read_raw(&m->S,name,t->q8,drop); } - else { if(t->fmt!=fmt||!t->q4){ t->fmt=fmt; t->O=O; t->I=I; t->q4=qalloc(nb); t->s=qsalloc(O); } st_read_raw(&m->S,name,t->q4,drop); } + int64_t ns=st_nbytes(&m->S,sn); /* scale bytes (F32) */ + /* Detect int4-grouped (fmt=4): packed int4 weight bytes BUT scale array is + * larger than O*4 — the group size is derived from the scale-array size. */ + int fmt = (nb==(int64_t)O*I)?1 : (nb==(int64_t)O*((I+1)/2))?2 : 3; + int gs=0; + if(fmt==2) gs=detect_group_size(O,I,ns); + if(gs>0) fmt=4; + if(fmt==1){ if(t->fmt!=1||!t->q8){ t->fmt=1; t->O=O; t->I=I; t->gs=0; t->q8=qalloc(nb); t->s=qsalloc(O); } st_read_raw(&m->S,name,t->q8,drop); } + else if(fmt==4){ int ng=(I+gs-1)/gs; + if(t->fmt!=4||!t->q4){ t->fmt=4; t->O=O; t->I=I; t->gs=gs; t->q4=qalloc(nb); t->s=falloc((int64_t)O*ng); } + st_read_raw(&m->S,name,t->q4,drop); } + else { if(t->fmt!=fmt||!t->q4){ t->fmt=fmt; t->O=O; t->I=I; t->gs=0; t->q4=qalloc(nb); t->s=qsalloc(O); } st_read_raw(&m->S,name,t->q4,drop); } st_read_f32(&m->S,sn,t->s,drop); } else { if(!t->qf && !t->q8 && !t->q4) qt_alloc(t,O,I,bits); @@ -1033,6 +1373,33 @@ static float *ld(Model *m, const char *name){ /* tensore 1D f32 residente (nor float *p=(float*)qalloc((size_t)n*sizeof(float)); /* registrato per la GPU sotto METAL */ st_read_f32(&m->S,name,p,0); return p; } +#ifdef COLI_CUDA +static void qt_cuda_colocate(QT *dst,const QT *src){ + if(!g_cuda_enabled||!g_cuda_dense||!dst->cuda_eligible||!src->cuda_eligible|| + dst->cuda_device==src->cuda_device)return; + int old=-1,now=-1;for(int i=0;icuda_device)old=i;if(g_cuda_devices[i]==src->cuda_device)now=i; + } + if(old>=0)g_cuda_dense_projected[old]-=qt_bytes(dst); + if(now>=0)g_cuda_dense_projected[now]+=qt_bytes(dst); + dst->cuda_device=src->cuda_device; +} +static void layer_cuda_shard_kvb(Layer *l,int H,int Q,int V){ + if(!g_cuda_enabled||!g_cuda_dense||g_cuda_ndev<2||l->kv_b.fmt==0)return; + int rb=l->kv_b.fmt==1?l->kv_b.I:(l->kv_b.fmt==2?(l->kv_b.I+1)/2:(l->kv_b.I+3)/4); + const uint8_t *weights=l->kv_b.fmt==1?(const uint8_t*)l->kv_b.q8:l->kv_b.q4; + for(int d=0,h0=0;dkv_b.s+(int64_t)h0*(Q+V); + if(!coli_cuda_tensor_upload(&l->kv_b_shard[d],part,scale,l->kv_b.fmt,l->kv_b.I,rows,g_cuda_devices[d]))return; + l->shard_h0[d]=h0;l->shard_hn[d]=hn;l->n_kv_b_shard++;h0+=hn; + } + int old=-1;for(int i=0;ikv_b.cuda_device)old=i; + if(old>=0)g_cuda_dense_projected[old]-=qt_bytes(&l->kv_b); + l->kv_b.cuda_eligible=0; +} +#endif static void model_init(Model *m, const char *snap, int cap, int ebits, int dbits){ memset(m,0,sizeof(*m)); m->ebits=ebits; m->dbits=dbits; @@ -1047,6 +1414,8 @@ static void model_init(Model *m, const char *snap, int cap, int ebits, int dbits m->L=calloc(c->n_layers,sizeof(Layer)); int NR=c->n_layers+1; /* +1: riga del layer MTP */ m->ecap=cap; m->ecache=calloc(NR,sizeof(ESlot*)); m->ecn=calloc(NR,sizeof(int)); + m->kv_dev_L=calloc(NR,sizeof(float*)); m->kv_dev_R=calloc(NR,sizeof(float*)); + m->kv_dev_valid=calloc(NR,sizeof(int)); m->eroute=calloc(NR,sizeof(int*)); m->enr=calloc(NR,sizeof(int)); m->pin=calloc(NR,sizeof(ESlot*)); m->npin=calloc(NR,sizeof(int)); m->eusage=calloc(NR,sizeof(uint32_t*)); m->eheat=calloc(NR,sizeof(uint32_t*)); @@ -1065,6 +1434,14 @@ static void model_init(Model *m, const char *snap, int cap, int ebits, int dbits l->kv_a_ln= ld(m,P("self_attn.kv_a_layernorm.weight")); l->kv_b = qt_load(m,P("self_attn.kv_b_proj.weight"), H*(c->qk_nope+c->v_head), c->kv_lora, dbits); l->o = qt_load(m,P("self_attn.o_proj.weight"), D, H*c->v_head, dbits); +#ifdef COLI_CUDA + qt_cuda_colocate(&l->o,&l->kv_b); + qt_cuda_colocate(&l->q_a,&l->kv_b); /* PIPE: intera catena attention sulla */ + qt_cuda_colocate(&l->q_b,&l->kv_b); /* stessa scheda / whole attention chain */ + qt_cuda_colocate(&l->kv_a,&l->kv_b); /* on the layer home device */ + if(getenv("COLI_CUDA_ATTN_SHARD")&&atoi(getenv("COLI_CUDA_ATTN_SHARD"))) + layer_cuda_shard_kvb(l,H,c->qk_nope,c->v_head); +#endif l->sparse = (i >= c->first_dense); if(!l->sparse){ l->gate_proj = qt_load(m,P("mlp.gate_proj.weight"), c->dense_inter, D, dbits); @@ -1077,6 +1454,11 @@ static void model_init(Model *m, const char *snap, int cap, int ebits, int dbits l->sh_gate = qt_load(m,P("mlp.shared_experts.gate_proj.weight"), sI, D, dbits); l->sh_up = qt_load(m,P("mlp.shared_experts.up_proj.weight"), sI, D, dbits); l->sh_down = qt_load(m,P("mlp.shared_experts.down_proj.weight"), D, sI, dbits); +#ifdef COLI_CUDA + qt_cuda_colocate(&l->sh_gate,&l->kv_b); /* PIPE2: shared chain on the layer home device */ + qt_cuda_colocate(&l->sh_up,&l->sh_gate); + qt_cuda_colocate(&l->sh_down,&l->sh_gate); +#endif m->ecache[i]=calloc(cap,sizeof(ESlot)); m->eroute[i]=calloc(c->topk,sizeof(int)); /* metodo C: ultimo routing del layer */ m->eusage[i]=calloc(c->n_experts,sizeof(uint32_t)); @@ -1199,11 +1581,19 @@ static void embed_row(Model *m, int tok, float *x){ static int g_mmap=0; static struct { int fd; void *base; size_t len; } g_maps[512]; static int g_nmaps; static pthread_mutex_t g_map_mtx = PTHREAD_MUTEX_INITIALIZER; /* expert_load e' OMP-parallel */ +/* forward decls: mem_should_wire/mem_wire live near pin_wire() further down, but + * qt_wire_mmap() (also further down, used by pin_wire()'s COLI_MMAP path) needs + * them declared before its own definition. Real mlock-ing of mmap'd pinned + * experts happens there, not in expert_load() -- see qt_wire_mmap() for why. */ +static int mem_should_wire(void); +static int mem_wire(void *addr, size_t len); +static void qt_unwire_mmap(QT *t); /* def. presso pin_wire / defined near pin_wire */ +static int64_t g_mmap_wired=0; static long g_mmap_wire_failed=0; static void *map_of_fd(int fd){ pthread_mutex_lock(&g_map_mtx); for(int i=0;ieid, so a * mispredicted cross-layer prefetch can never kill the server. */ +/* pread completo: gestisce le short-read (POSIX le ammette su file regolari + * sotto pressione di memoria) e le EINTR, e riporta un errore ONESTO. perror + * stampava "Success" quando pread ritorna un conteggio corto invece di -1 + * (errno resta 0 dalla syscall precedente) -> messaggio fuorviante nel path + * score/bench (#236). Ritorna 0 = ok, -1 = errore reale o EOF. */ +static int pread_full(int fd, void *buf, int64_t n, int64_t off, const char *tag){ + char *p=buf; int64_t got=0; + while(gotS,qn); if(!tw[k]||!tq[k]){ fprintf(stderr,"missing %s\n",nm[k]); if(fatal) exit(1); return -1; } } + if(g_disk_split){ /* split load/byte per tipo layer; atomici: expert_load gira anche su OMP/pipe/pilot */ + int64_t tb=0; for(int k=0;k<3;k++) tb+=tw[k]->nbytes+tq[k]->nbytes; + if(layer==c->n_layers){ __atomic_add_fetch(&m->ld_mtp,1,__ATOMIC_RELAXED); + __atomic_add_fetch(&m->bytes_mtp,(uint64_t)tb,__ATOMIC_RELAXED); } + else { __atomic_add_fetch(&m->ld_main,1,__ATOMIC_RELAXED); + __atomic_add_fetch(&m->bytes_main,(uint64_t)tb,__ATOMIC_RELAXED); } + } if(g_mmap){ void *bw[3],*bq[3]; int okm=1; for(int k=0;k<3;k++){ @@ -1265,7 +1680,11 @@ static int expert_load(Model *m, int layer, int eid, ESlot *s, int fatal){ for(int k=0;k<3;k++){ int64_t nb=tw[k]->nbytes; int fmt=(nb==(int64_t)OO[k]*II[k])?1:(nb==(int64_t)OO[k]*((II[k]+1)/2))?2:3; - qt[k]->fmt=fmt; qt[k]->O=OO[k]; qt[k]->I=II[k]; qt[k]->qf=NULL; + /* detect grouped int4 (fmt=4): int4 weight bytes + larger scale array */ + int gs=0; + if(fmt==2) gs=detect_group_size(OO[k],II[k],tq[k]->nbytes); + if(gs>0) fmt=4; + qt[k]->fmt=fmt; qt[k]->O=OO[k]; qt[k]->I=II[k]; qt[k]->gs=gs; qt[k]->qf=NULL; qt[k]->q8=(int8_t*)((char*)bw[k]+tw[k]->off); qt[k]->q4=(uint8_t*)((char*)bw[k]+tw[k]->off); qt[k]->s=(float*)((char*)bq[k]+tq[k]->off); } @@ -1275,7 +1694,7 @@ static int expert_load(Model *m, int layer, int eid, ESlot *s, int fatal){ * residency. This is pread's I/O without the copy and without the slab. */ for(int k=0;k<3;k++){ char *p=(char*)bw[k]+tw[k]->off; size_t n=(size_t)tw[k]->nbytes; -#if defined(__APPLE__) || defined(__linux__) +#if defined(__APPLE__) || defined(__linux__) || defined(__FreeBSD__) madvise((void*)((uintptr_t)p & ~16383UL), n+16384, MADV_WILLNEED); #endif volatile char acc=0; @@ -1283,6 +1702,13 @@ static int expert_load(Model *m, int layer, int eid, ESlot *s, int fatal){ acc+=p[n-1]; (void)acc; char *q=(char*)bq[k]+tq[k]->off; size_t nq=(size_t)tq[k]->nbytes; for(size_t i=0;islab, always NULL under mmap, so wiring here would + * leak locked pages for every GPU-tier expert). See pin_wire() below: it wires + * the final resident set only, after GPU release has already nulled out the + * pointers for anything that isn't genuinely RAM-tier. */ } s->eid=eid; return 0; } @@ -1356,19 +1782,19 @@ static int expert_load(Model *m, int layer, int eid, ESlot *s, int fatal){ } } if(!done){ /* fallback bufferizzato */ - if(pread(tw[ord[0]]->fd, s->slab, wtot, off0)!=wtot){ perror("pread expert"); if(fatal) exit(1); return -1; } + if(pread_full(tw[ord[0]]->fd, s->slab, wtot, off0, "pread expert")){ if(fatal) exit(1); return -1; } pos[ord[0]]=0; pos[ord[1]]=tw[ord[0]]->nbytes; pos[ord[2]]=tw[ord[0]]->nbytes+tw[ord[1]]->nbytes; done=1; } } if(!done){ /* non contigui: 3 pread bufferizzate */ int64_t o=0; for(int a=0;a<3;a++){ int k=ord[a]; - if(pread(tw[k]->fd, s->slab+o, tw[k]->nbytes, tw[k]->off)!=tw[k]->nbytes){ perror("pread expert"); if(fatal) exit(1); return -1; } + if(pread_full(tw[k]->fd, s->slab+o, tw[k]->nbytes, tw[k]->off, "pread expert")){ if(fatal) exit(1); return -1; } pos[k]=o; o+=tw[k]->nbytes; } } float *fp[3]; int64_t fo=0; /* scale (piccole) */ for(int k=0;k<3;k++){ - if(pread(tq[k]->fd, (char*)(s->fslab+fo), tq[k]->nbytes, tq[k]->off)!=tq[k]->nbytes){ perror("pread qs"); if(fatal) exit(1); return -1; } + if(pread_full(tq[k]->fd, (char*)(s->fslab+fo), tq[k]->nbytes, tq[k]->off, "pread qs")){ if(fatal) exit(1); return -1; } fp[k]=s->fslab+fo; fo+=tq[k]->nbytes/4; } if(g_drop){ /* scarta subito le pagine: evita che la page * cache in pressione strangoli il throughput */ @@ -1379,12 +1805,198 @@ static int expert_load(Model *m, int layer, int eid, ESlot *s, int fatal){ for(int k=0;k<3;k++){ int64_t nb=tw[k]->nbytes; int fmt = (nb==(int64_t)OO[k]*II[k])?1 : (nb==(int64_t)OO[k]*((II[k]+1)/2))?2 : 3; - qt[k]->fmt=fmt; qt[k]->O=OO[k]; qt[k]->I=II[k]; qt[k]->qf=NULL; + int gs=0; + if(fmt==2) gs=detect_group_size(OO[k],II[k],tq[k]->nbytes); + if(gs>0) fmt=4; + qt[k]->fmt=fmt; qt[k]->O=OO[k]; qt[k]->I=II[k]; qt[k]->gs=gs; qt[k]->qf=NULL; qt[k]->q8=(int8_t*)(s->slab+pos[k]); qt[k]->q4=s->slab+pos[k]; qt[k]->s=fp[k]; } s->eid=eid; return 0; } +#ifdef __linux__ +/* io_uring expert batches. One owner prepares all reads for a block, submits + * them in one syscall, and reaps CQEs on demand. The kernel, rather than a set + * of blocking pthreads, owns the I/O concurrency. */ +#define URING_LOAD_MAX 64 +#define URING_REQ_MAX 512 +typedef struct { + int load, expect; +} UringRead; +typedef struct { + Model *m; ESlot *s; int layer,eid,fatal; + st_tensor *tw[3],*tq[3]; int64_t pos[3]; + int pending,done,finalized,error; +} UringLoad; +typedef struct { + ColiUring ring; + UringLoad load[URING_LOAD_MAX]; + UringRead req[URING_REQ_MAX]; + int nload,nreq,started; +} UringBatch; +static UringBatch g_ub_pipe, g_ub_pilot; + +static int uring_batch_init(UringBatch *b){ + if(b->started) return 0; + if(coli_uring_init(&b->ring,URING_REQ_MAX)) return -1; + b->started=1; return 0; +} +static void uring_batch_reset(UringBatch *b){ + b->nload=0; b->nreq=0; +} +static int uring_load_error(UringLoad *l,int err,const char *what){ + l->error=err?err:EIO; l->done=1; + if(l->fatal){ errno=l->error; perror(what); exit(1); } + return -1; +} +static int uring_add_read(UringBatch *b,int li,int fd,void *buf,size_t len, + int64_t off,size_t expect){ + if(b->nreq>=URING_REQ_MAX || expect>INT_MAX){ errno=E2BIG; return -1; } + int ri=b->nreq++; + b->req[ri]=(UringRead){li,(int)expect}; + if(coli_uring_prep_read(&b->ring,fd,buf,len,off,(uint64_t)ri+1)) return -1; + b->load[li].pending++; + return 0; +} +/* Returns the load index. URING is intentionally a quantized streaming path; + * unsupported layouts fail instead of silently dropping back to pread. */ +static int uring_load_add(UringBatch *b,Model *m,int layer,int eid,ESlot *s,int fatal){ + if(b->nload>=URING_LOAD_MAX){ errno=E2BIG; return -1; } + int li=b->nload++; + UringLoad *l=&b->load[li]; memset(l,0,sizeof(*l)); + l->m=m; l->s=s; l->layer=layer; l->eid=eid; l->fatal=fatal; + char nm[3][288],qn[300]; const char *suf[3]={"gate_proj","up_proj","down_proj"}; + for(int k=0;k<3;k++) snprintf(nm[k],sizeof(nm[k]),"model.layers.%d.mlp.experts.%d.%s.weight",layer,eid,suf[k]); + snprintf(qn,sizeof(qn),"%s.qs",nm[0]); + if(g_mmap || !st_has(&m->S,qn)) + return uring_load_error(l,ENOTSUP,"URING requires quantized expert tensors"),li; +#ifdef COLI_CUDA + if(s->eid!=eid){ qt_cuda_reset(&s->g); qt_cuda_reset(&s->u); qt_cuda_reset(&s->d); } +#endif + for(int k=0;k<3;k++){ + l->tw[k]=st_find(&m->S,nm[k]); + size_t n=strnlen(nm[k],sizeof(nm[k])); + if(n+3>=sizeof(qn)) return uring_load_error(l,ENAMETOOLONG,"io_uring expert metadata"),li; + memcpy(qn,nm[k],n); memcpy(qn+n,".qs",4); l->tq[k]=st_find(&m->S,qn); + if(!l->tw[k]||!l->tq[k]) return uring_load_error(l,ENOENT,"io_uring expert metadata"),li; + } + int64_t wtot=l->tw[0]->nbytes+l->tw[1]->nbytes+l->tw[2]->nbytes; + int64_t ftot=(l->tq[0]->nbytes+l->tq[1]->nbytes+l->tq[2]->nbytes)/4; + if(wtot<=0 || ftot<=0) return uring_load_error(l,EINVAL,"io_uring expert size"),li; + if(!s->slab || wtot+8192>s->slab_cap){ +#ifdef COLI_METAL + if(s->slab&&g_metal_enabled) coli_metal_unregister(s->slab); + compat_aligned_free(s->slab); + size_t need=((size_t)wtot+8192+16383)&~(size_t)16383; + if(posix_memalign((void**)&s->slab,16384,need)){ + s->slab=NULL; s->slab_cap=0; return uring_load_error(l,ENOMEM,"io_uring expert slab"),li; } + s->slab_cap=need; if(g_metal_enabled) coli_metal_register(s->slab,need); +#else + compat_aligned_free(s->slab); + if(posix_memalign((void**)&s->slab,4096,(size_t)wtot+8192)){ + s->slab=NULL; s->slab_cap=0; return uring_load_error(l,ENOMEM,"io_uring expert slab"),li; } + s->slab_cap=wtot+8192; +#endif + } + if(!s->fslab || ftot>s->fslab_cap){ +#ifdef COLI_METAL + if(s->fslab&&g_metal_enabled) coli_metal_unregister(s->fslab); + free(s->fslab); size_t fb=(((size_t)ftot*sizeof(float))+16383)&~(size_t)16383; + if(posix_memalign((void**)&s->fslab,16384,fb)){ + s->fslab=NULL; s->fslab_cap=0; return uring_load_error(l,ENOMEM,"io_uring expert scales"),li; } + s->fslab_cap=ftot; if(g_metal_enabled) coli_metal_register(s->fslab,fb); +#else + free(s->fslab); s->fslab=malloc((size_t)ftot*sizeof(float)); + if(!s->fslab){ s->fslab_cap=0; return uring_load_error(l,ENOMEM,"io_uring expert scales"),li; } + s->fslab_cap=ftot; +#endif + } + int ord[3]={0,1,2}; + for(int a=0;a<3;a++) for(int z=a+1;z<3;z++) if(l->tw[ord[z]]->offtw[ord[a]]->off){int t=ord[a];ord[a]=ord[z];ord[z]=t;} + int contig=l->tw[ord[0]]->fd==l->tw[ord[1]]->fd && l->tw[ord[1]]->fd==l->tw[ord[2]]->fd + && l->tw[ord[0]]->off+l->tw[ord[0]]->nbytes==l->tw[ord[1]]->off + && l->tw[ord[1]]->off+l->tw[ord[1]]->nbytes==l->tw[ord[2]]->off; + if(contig){ + int64_t off0=l->tw[ord[0]]->off; + int dfd=g_direct?st_direct_fd(&m->S,l->tw[ord[0]]->fd):-1; + if(dfd>=0){ + int64_t base=off0&~4095LL,need=(off0-base)+wtot,len=(need+4095)&~4095LL; + l->pos[ord[0]]=off0-base; l->pos[ord[1]]=l->pos[ord[0]]+l->tw[ord[0]]->nbytes; + l->pos[ord[2]]=l->pos[ord[1]]+l->tw[ord[1]]->nbytes; + if(uring_add_read(b,li,dfd,s->slab,(size_t)len,base,(size_t)need)) + return uring_load_error(l,errno,"io_uring direct expert read"),li; + }else{ + l->pos[ord[0]]=0; l->pos[ord[1]]=l->tw[ord[0]]->nbytes; + l->pos[ord[2]]=l->pos[ord[1]]+l->tw[ord[1]]->nbytes; + if(uring_add_read(b,li,l->tw[ord[0]]->fd,s->slab,(size_t)wtot,off0,(size_t)wtot)) + return uring_load_error(l,errno,"io_uring expert read"),li; + } + }else{ + int64_t o=0; + for(int a=0;a<3;a++){ int k=ord[a]; l->pos[k]=o; + if(uring_add_read(b,li,l->tw[k]->fd,s->slab+o,(size_t)l->tw[k]->nbytes,l->tw[k]->off,(size_t)l->tw[k]->nbytes)) + return uring_load_error(l,errno,"io_uring expert read"),li; + o+=l->tw[k]->nbytes; + } + } + int64_t fo=0; + for(int k=0;k<3;k++){ + if(uring_add_read(b,li,l->tq[k]->fd,s->fslab+fo,(size_t)l->tq[k]->nbytes,l->tq[k]->off,(size_t)l->tq[k]->nbytes)) + return uring_load_error(l,errno,"io_uring expert scale read"),li; + fo+=l->tq[k]->nbytes/4; + } + return li; +} +static void uring_reap(UringBatch *b){ + struct io_uring_cqe cqe; + while(coli_uring_peek(&b->ring,&cqe)){ + if(!cqe.user_data || cqe.user_data>(uint64_t)b->nreq) continue; + UringRead *r=&b->req[cqe.user_data-1]; UringLoad *l=&b->load[r->load]; + if(cqe.resexpect && !l->error) l->error=cqe.res<0?-cqe.res:EIO; + if(l->pending>0) l->pending--; + if(l->pending==0) l->done=1; + } +} +static int uring_submit_batch(UringBatch *b){ + if(coli_uring_enter(&b->ring,0)<0) return -1; + uring_reap(b); return 0; +} +static int uring_wait_load(UringBatch *b,int li){ + UringLoad *l=&b->load[li]; + while(!l->done){ + uring_reap(b); if(l->done) break; + if(coli_uring_enter(&b->ring,1)<0) return uring_load_error(l,errno,"io_uring wait"); + } + return l->error?-1:0; +} +static int uring_finalize_load(UringBatch *b,int li,int publish_eid){ + UringLoad *l=&b->load[li]; ESlot *s=l->s; + if(l->finalized) return 0; + if(uring_wait_load(b,li)<0){ errno=l->error; if(l->fatal){perror("io_uring expert completion");exit(1);} return -1; } + if(g_drop){ + int ord0=0; for(int k=1;k<3;k++) if(l->tw[k]->offtw[ord0]->off) ord0=k; + int64_t wtot=l->tw[0]->nbytes+l->tw[1]->nbytes+l->tw[2]->nbytes; + posix_fadvise(l->tw[ord0]->fd,l->tw[ord0]->off,wtot,POSIX_FADV_DONTNEED); + for(int k=0;k<3;k++) posix_fadvise(l->tq[k]->fd,l->tq[k]->off,l->tq[k]->nbytes,POSIX_FADV_DONTNEED); + } + Cfg *c=&l->m->c; int I=c->moe_inter,D=c->hidden; float *fp[3]; int64_t fo=0; + QT *qt[3]={&s->g,&s->u,&s->d}; int OO[3]={I,I,D},II[3]={D,D,I}; + for(int k=0;k<3;k++){ + fp[k]=s->fslab+fo; fo+=l->tq[k]->nbytes/4; + int64_t nb=l->tw[k]->nbytes; + int fmt=(nb==(int64_t)OO[k]*II[k])?1:(nb==(int64_t)OO[k]*((II[k]+1)/2))?2:3; + qt[k]->fmt=fmt; qt[k]->O=OO[k]; qt[k]->I=II[k]; qt[k]->qf=NULL; + qt[k]->q8=(int8_t*)(s->slab+l->pos[k]); qt[k]->q4=s->slab+l->pos[k]; qt[k]->s=fp[k]; + } + if(publish_eid) s->eid=l->eid; + l->finalized=1; return 0; +} +static int uring_wait_all(UringBatch *b){ + for(int i=0;inload;i++) if(uring_wait_load(b,i)<0) return -1; + return 0; +} +#endif + /* ============================ PIPE: load ‖ matmul ============================ * Overlap NVMe expert-weight loads with expert matmul. A small persistent pool * of I/O worker pthreads runs the misses' pread (expert_load) into distinct @@ -1412,8 +2024,12 @@ static int expert_load(Model *m, int layer, int eid, ESlot *s, int fatal){ * condvar exist ONLY to park/wake idle workers, never for correctness. Gated * behind PIPE=1; OFF => the original blocking-load + serial-matmul path runs * byte-identically. */ -static int g_pipe=0; /* PIPE=1: async expert-load pipeline (default OFF) */ +static int g_pipe=0; /* PIPE=1: async expert-load pipeline. Default ON for Windows + * (parsed in main: getenv("PIPE")?:1 on _WIN32, :0 elsewhere). + * Keeps expert pread off the forward-pass thread so loads overlap + * the matmul. PIPE=0 opts back into the blocking serial path. */ static int g_pipe_nw=8; /* PIPE_WORKERS=n: I/O worker threads (disk-parallel reads) */ +static int g_uring=0; /* URING=1: Linux io_uring load/completion backend; implies PIPE */ typedef struct { _Atomic uint64_t cur; /* (gen<<8)|index; gen main-only, index 0..njobs (≤64) */ _Atomic int njobs; /* current batch job count */ @@ -1453,6 +2069,12 @@ static void *pipe_worker(void *arg){ } static void pipe_init(Model *m){ if(g_pp.started) return; +#ifdef __linux__ + if(g_uring){ + if(uring_batch_init(&g_ub_pipe)){ perror("URING=1 io_uring_setup"); exit(1); } + g_pp.m=m; g_pp.started=1; return; + } +#endif g_pp.m=m; g_pp.nw=g_pipe_nw; if(g_pp.nw>16) g_pp.nw=16; if(g_pp.nw<1) g_pp.nw=1; atomic_store(&g_pp.cur,0); atomic_store(&g_pp.njobs,0); pthread_mutex_init(&g_pp.mx,NULL); pthread_cond_init(&g_pp.cv,NULL); @@ -1463,6 +2085,17 @@ static void pipe_init(Model *m){ * Order is load-bearing: write all batch state RELAXED, then RELEASE-store cur to * publish it, then wake parked workers. */ static void pipe_dispatch(Model *m,int layer,const int *eids,int njobs){ +#ifdef __linux__ + if(g_uring){ + uring_batch_reset(&g_ub_pipe); + for(int q=0;qws[q],1); + if(li!=q){ fprintf(stderr,"URING: expert batch overflow\n"); exit(1); } + } + if(uring_submit_batch(&g_ub_pipe)){ perror("URING: submit"); exit(1); } + return; + } +#endif g_pp.m=m; atomic_store_explicit(&g_pp.njobs,njobs,memory_order_relaxed); atomic_store_explicit(&g_pp.layer,layer,memory_order_relaxed); @@ -1473,13 +2106,19 @@ static void pipe_dispatch(Model *m,int layer,const int *eids,int njobs){ pthread_mutex_lock(&g_pp.mx); pthread_cond_broadcast(&g_pp.cv); pthread_mutex_unlock(&g_pp.mx); } static inline void pipe_wait(int q){ +#ifdef __linux__ + if(g_uring){ + if(uring_finalize_load(&g_ub_pipe,q,1)){ perror("URING: expert load"); exit(1); } + return; + } +#endif while(!atomic_load_explicit(&g_pp.ready[q],memory_order_acquire)) sched_yield(); } #ifdef COLI_CUDA static void expert_host_release(Model *m, ESlot *s){ if(!s->slab&&!s->fslab) return; -#if defined(__APPLE__) || defined(__linux__) +#if defined(__APPLE__) || defined(__linux__) || defined(__FreeBSD__) if(s->slab) munlock(s->slab,(size_t)s->slab_cap); if(s->fslab) munlock(s->fslab,(size_t)s->fslab_cap*sizeof(float)); #elif defined(_WIN32) @@ -1487,7 +2126,12 @@ static void expert_host_release(Model *m, ESlot *s){ if(s->fslab) compat_munlock(s->fslab,(size_t)s->fslab_cap*sizeof(float)); #endif int64_t bytes=qt_bytes(&s->g)+qt_bytes(&s->u)+qt_bytes(&s->d); - free(s->slab); free(s->fslab); s->slab=NULL; s->fslab=NULL; s->slab_cap=s->fslab_cap=0; + /* slab is posix_memalign'd: on Windows that is _aligned_malloc, and plain + * free() corrupts the CRT heap (0xC0000374) — same bug the compat.h audit + * fixed at the original expert_load site. fslab is plain malloc/falloc + * on the CPU path, so its free() stays plain (Metal path frees it before + * re-alloc and never reaches here with an aligned fslab on _WIN32). */ + compat_aligned_free(s->slab); free(s->fslab); s->slab=NULL; s->fslab=NULL; s->slab_cap=s->fslab_cap=0; QT *q[3]={&s->g,&s->u,&s->d}; for(int k=0;k<3;k++){ q[k]->qf=NULL; q[k]->q8=NULL; q[k]->q4=NULL; q[k]->s=NULL; } m->resident_bytes-=bytes; if(m->resident_bytes<0) m->resident_bytes=0; @@ -1536,7 +2180,10 @@ static void qt_matvec_rows(const QT *t, int r0, int n, const float *x, float *y) y[j]=(float)a; } } -static int g_absorb=-1; /* ABSORB: -1 auto (decode S<=4), 0 mai, 1 sempre (test) */ +static int g_absorb=-1; +#ifdef COLI_CUDA +static int g_cuda_pipe=0; /* COLI_CUDA_PIPE=1: prefill attention chain resident on the layer home device */ +#endif /* ABSORB: -1 auto (decode S<=4), 0 mai, 1 sempre (test) */ static int g_dsa_force=0; /* DSA_FORCE=1: selezione sempre attiva (test: top-min(k,T)=denso) */ static int cmp_fdesc(const void *a,const void *b){ float x=*(const float*)a, y=*(const float*)b; return xy?-1:0; } @@ -1544,6 +2191,141 @@ static int cmp_fdesc(const void *a,const void *b){ /* attenzione MLA con KV-cache compressa, su token nuovi x[S,hidden], pos_base = pos del primo */ /* kvs/pos describe a ragged decode batch: each row may belong to a different * sequence. NULL keeps the original contiguous, currently-bound KV path. */ +#ifdef COLI_CUDA +/* Ombra KV su device per il DECODE: righe [0,upto) valide sulla scheda di kv_b. + * L'host resta canonico; l'ombra si riallinea in blocco quando resta indietro e + * viene invalidata da kv_bind / dalla riscrittura di righe gia' specchiate. */ +static int kv_dev_sync(Model *m, Layer *l, int layer, int upto){ + Cfg *c=&m->c; int kvl=c->kv_lora, R=c->qk_rope, dev=l->kv_b.cuda_device; + if(upto>m->max_t) return 0; + if(!m->kv_dev_L[layer]){ + m->kv_dev_L[layer]=(float*)coli_cuda_pipe_alloc(dev,(size_t)m->max_t*kvl*4); + m->kv_dev_R[layer]=(float*)coli_cuda_pipe_alloc(dev,(size_t)m->max_t*R*4); + m->kv_dev_valid[layer]=0; + if(!m->kv_dev_L[layer]||!m->kv_dev_R[layer]) return 0; + } + int v=m->kv_dev_valid[layer]; + if(vkv_dev_L[layer]+(size_t)v*kvl, + coli_kv_row(m->kv->Lc[layer],v,kvl),(size_t)(upto-v)*kvl*4)|| + !coli_cuda_pipe_upload(dev,m->kv_dev_R[layer]+(size_t)v*R, + coli_kv_row(m->kv->Rc[layer],v,R),(size_t)(upto-v)*R*4)) return 0; + m->kv_dev_valid[layer]=upto; + } + return 1; +} + +/* Inc.1a — catena attention residente sul device del layer / attention chain + * resident on the layer home device. Proiezioni q/kv, norme, RoPE, batch + * attention e o_proj girano sulla scheda di kv_b; scaricano solo out [S,D], + * i nuovi record KV [S,kvl+R] e nulla altro. Ritorna 0 su qualsiasi errore: + * il chiamante riesegue il percorso CPU (idempotente). */ +static int attn_pipe_prefill(Model *m, Layer *l, int layer, const float *x, int x_is_dev, + int S, int pos_base, float *out, float *out_dev){ + Cfg *c=&m->c; int H=c->n_heads, D=c->hidden, qh=c->qk_head; + int kvl=c->kv_lora, R=c->qk_rope, ql=c->q_lora; + int dev=l->kv_b.cuda_device; + if(l->q_a.cuda_device!=dev||l->q_b.cuda_device!=dev|| + l->kv_a.cuda_device!=dev||l->o.cuda_device!=dev) return 0; + int st0=m->kv_start[layer], T=pos_base+S-st0, old=pos_base-st0; + if(T8192) return 0; + double t0=now_s(); + size_t xb=(size_t)S*D*4, qrb=(size_t)S*ql*4, qb=(size_t)S*H*qh*4; + size_t cb=(size_t)S*(kvl+R)*4, lb=(size_t)T*kvl*4, rb=(size_t)T*R*4; + float *chost=NULL; int ok=0; + /* scratch persistenti (slot fissi per device): zero churn di cudaMalloc */ + float *xd =x_is_dev?(float*)x:coli_cuda_pipe_scratch(dev,0,xb); + float *qrd=coli_cuda_pipe_scratch(dev,1,qrb); + float *qd =coli_cuda_pipe_scratch(dev,2,qb), *cd =coli_cuda_pipe_scratch(dev,3,cb); + float *ld_=coli_cuda_pipe_scratch(dev,4,lb), *rd =coli_cuda_pipe_scratch(dev,5,rb); + float *w1 =coli_cuda_pipe_scratch(dev,6,(size_t)ql*4); + float *w2 =coli_cuda_pipe_scratch(dev,7,(size_t)kvl*4); + chost=(float*)malloc(cb); + if(!xd||!qrd||!qd||!cd||!ld_||!rd||!w1||!w2||!chost) goto done; + if((!x_is_dev&&!coli_cuda_pipe_upload(dev,xd,x,xb))|| + !coli_cuda_pipe_upload(dev,w1,l->q_a_ln,(size_t)ql*4)|| + !coli_cuda_pipe_upload(dev,w2,l->kv_a_ln,(size_t)kvl*4)) goto done; + /* proiezioni + norme + rope, tutto sul device */ + if(!coli_cuda_pipe_gemm(l->q_a.cuda,qrd,xd,S)) goto done; + if(!coli_cuda_pipe_rmsnorm(dev,qrd,qrd,w1,S,ql,c->eps)) goto done; + if(!coli_cuda_pipe_gemm(l->q_b.cuda,qd,qrd,S)) goto done; + if(!coli_cuda_pipe_rope_base(dev,qd,pos_base,S*H,qh,c->qk_nope,R,H,c->theta)) goto done; + if(!coli_cuda_pipe_gemm(l->kv_a.cuda,cd,xd,S)) goto done; + if(!coli_cuda_pipe_rmsnorm_s(dev,cd,cd,w2,S,kvl,c->eps,kvl+R,kvl+R)) goto done; + if(!coli_cuda_pipe_rope_base(dev,cd,pos_base,S,kvl+R,kvl,R,1,c->theta)) goto done; + /* cache latente [T,kvl] + rot [T,R] contigue: righe vecchie da host, nuove da cd */ + if(old>0){ + if(!coli_cuda_pipe_upload(dev,ld_,coli_kv_row(m->Lc[layer],st0,kvl),(size_t)old*kvl*4)|| + !coli_cuda_pipe_upload(dev,rd,coli_kv_row(m->Rc[layer],st0,R),(size_t)old*R*4)) goto done; + } + if(!coli_cuda_pipe_copy2d(dev,ld_+(size_t)old*kvl,kvl,cd,kvl+R,kvl,S)) goto done; + if(!coli_cuda_pipe_copy2d(dev,rd+(size_t)old*R,R,cd+kvl,kvl+R,R,S)) goto done; + /* KV host resta canonica: scarica i record nuovi (gia' normati+ropati) */ + if(!coli_cuda_pipe_download(dev,cd,chost,cb)) goto done; + for(int s=0;sLc[layer],pos_base+s,kvl),chost+(size_t)s*(kvl+R),kvl*4); + memcpy(coli_kv_row(m->Rc[layer],pos_base+s,R),chost+(size_t)s*(kvl+R)+kvl,R*4); + } + if(m->kv_dev_valid[layer]>pos_base) m->kv_dev_valid[layer]=pos_base; + m->t_aproj+=now_s()-t0; t0=now_s(); +#ifdef COLI_CUDA + /* Negativo (2026-07-13): P2P a stella dal device di casa serializza ~95MB/layer + * sul suo link PCIe — attention 26->41-44s. Resta opt-in per topologie NVLink. */ + if(out_dev && l->n_kv_b_shard>1 && + getenv("COLI_CUDA_PIPE_SHARD") && atoi(getenv("COLI_CUDA_PIPE_SHARD"))){ + /* head-shard nel pipeline: q gia' sul device di casa. Per ogni scheda: + * slice di q (repack strided->contiguo), broadcast latent+rope via P2P, + * score parallelo sui rispettivi head, ctx slice riportata a casa e + * ricomposta, poi o_proj residente. */ + int n=l->n_kv_b_shard, vh=c->v_head; + size_t ctxb=(size_t)S*H*vh*4; + size_t stage_one=(size_t)S*H*(size_t)(c->qk_head>vh?c->qk_head:vh)*4; + float *ctx_full=coli_cuda_pipe_scratch(dev,16,ctxb); + float *stage=coli_cuda_pipe_scratch(dev,17,stage_one*n); + int ok_sh=(ctx_full&&stage)?1:0; + if(ok_sh){ + #pragma omp parallel for schedule(static) reduction(&:ok_sh) + for(int d2=0;d2shard_hn[d2], h0=l->shard_h0[d2]; + int sdev=coli_cuda_tensor_device(l->kv_b_shard[d2]); + float *st=stage+(size_t)d2*(stage_one/4); + size_t qsb=(size_t)S*hn*qh*4, csb=(size_t)S*hn*vh*4; + float *qs_r=coli_cuda_pipe_scratch(sdev,18,qsb); + float *ld_r=coli_cuda_pipe_scratch(sdev,19,(size_t)T*kvl*4); + float *rr_r=coli_cuda_pipe_scratch(sdev,20,(size_t)T*R*4); + float *cx_r=coli_cuda_pipe_scratch(sdev,21,csb); + int okd=qs_r&&ld_r&&rr_r&&cx_r; + /* slice di q: [S,H,qh] -> [S,hn,qh] contigua sul device di casa */ + okd=okd&&coli_cuda_pipe_copy2d(dev,st,hn*qh,qd+(size_t)h0*qh,H*qh,hn*qh,S); + okd=okd&&coli_cuda_pipe_peer_copy(sdev,qs_r,dev,st,qsb); + okd=okd&&coli_cuda_pipe_peer_copy(sdev,ld_r,dev,ld_,(size_t)T*kvl*4); + okd=okd&&coli_cuda_pipe_peer_copy(sdev,rr_r,dev,rd,(size_t)T*R*4); + okd=okd&&coli_cuda_attention_absorb_batch_dev(l->kv_b_shard[d2],cx_r,qs_r,ld_r,rr_r, + S,hn,c->qk_nope,R,vh,kvl,T,c->attn_scale); + okd=okd&&coli_cuda_pipe_peer_copy(dev,st,sdev,cx_r,csb); + okd=okd&&coli_cuda_pipe_copy2d(dev,ctx_full+(size_t)h0*vh,H*vh,st,hn*vh,hn*vh,S); + ok_sh&=okd; + } + } + if(ok_sh){ + ok=coli_cuda_pipe_gemm(l->o.cuda,out_dev,ctx_full,S)&&coli_cuda_pipe_sync(dev); + } else ok=0; + if(!ok) + ok=coli_cuda_attention_project_batch_dev_out(l->kv_b.cuda,l->o.cuda,out_dev,qd,ld_,rd, + S,H,c->qk_nope,R,c->v_head,kvl,T,c->attn_scale); + } else +#endif + ok=out_dev?coli_cuda_attention_project_batch_dev_out(l->kv_b.cuda,l->o.cuda,out_dev,qd,ld_,rd, + S,H,c->qk_nope,R,c->v_head,kvl,T,c->attn_scale) + :coli_cuda_attention_project_batch_dev(l->kv_b.cuda,l->o.cuda,out,qd,ld_,rd, + S,H,c->qk_nope,R,c->v_head,kvl,T,c->attn_scale); + m->t_acore+=now_s()-t0; +done: + free(chost); /* gli scratch device restano al contesto */ + return ok; +} +#endif + static void attention_rows(Model *m, Layer *l, int layer, float *x, int S, int pos_base, KVState *const *kvs, const int *positions, float *out){ Cfg *c=&m->c; int H=c->n_heads, D=c->hidden, qh=c->qk_head, vh=c->v_head; @@ -1551,8 +2333,14 @@ static void attention_rows(Model *m, Layer *l, int layer, float *x, int S, int p double ta0=now_s(); #ifdef COLI_METAL /* Fused decode attention on GPU: whole layer in one command buffer (keeps the GPU hot). - * S<=4 absorption path with st0==0, DSA selection inactive, and GLM-5.2 int4 dims. */ - if(g_metal_enabled && S<=4 && (g_absorb==1||(g_absorb<0&&S<=4)) && m->kv_start[layer]==0 + * S<=4 absorption path with st0==0, DSA selection inactive, and GLM-5.2 int4 dims. + * RAGGED GUARD (!kvs): the kernel takes ONE Lc/Rc pair and ONE pos_base — it assumes + * row s is token pos_base+s of the SAME sequence. The batched mux decode + * (step_decode_batch) passes per-row kvs[]/positions[] with pos_base=0, so the kernel + * would rope every row at position 0 and attend over a 1-token window of the wrong + * cache -> greedy decode hits EOS at token 2 (mux answers truncated to 1 token). + * Ragged rows take the CPU absorb path below, which reads kvs[s]/positions[s]. */ + if(g_metal_enabled && !kvs && S<=4 && (g_absorb==1||(g_absorb<0&&S<=4)) && m->kv_start[layer]==0 && D==6144 && H==64 && c->q_lora==2048 && c->kv_lora==512 && c->qk_nope==192 && c->qk_rope==64 && vh==256 && l->kv_b.fmt==2){ int sel_active = m->has_dsa && layern_layers && c->idx_type[layer] && (pos_base+S) > c->index_topk; @@ -1578,23 +2366,54 @@ static void attention_rows(Model *m, Layer *l, int layer, float *x, int S, int p #endif float *ctx=falloc((int64_t)S*H*vh); float *Q=falloc((int64_t)S*H*qh); /* query (roped) dei token nuovi */ - float *QR=falloc((int64_t)S*c->q_lora), *comp=falloc(c->kv_lora+c->qk_rope); - /* 1) per ogni token nuovo: query roped + latente normato e k_rot roped -> in cache. - * QR tiene il residuo q_a per TUTTE le posizioni: serve anche all'indexer DSA. */ - for(int s=0;skv_lora+c->qk_rope; + float *QR=falloc((int64_t)S*c->q_lora), *comp=falloc((int64_t)S*cw); + /* 1) query roped + latente normato e k_rot roped -> in cache. + * QR tiene il residuo q_a per TUTTE le posizioni: serve anche all'indexer DSA. + * + * BATCH-ROWS: le tre proiezioni girano su tutte le S righe in un colpo solo, come gia' fa + * o_proj (matmul_qt(...,S) sotto) e come fa moe() con la batch-union. Una riga per volta + * il peso veniva ri-letto per OGNI token; a S righe si legge una volta sola. + * matmul_qt_ex(...,0): restano sul kernel int4 ESATTO. Con l'IDOT (che il gate S>=g_i4s + * abiliterebbe da solo appena S>1) il prefill sarebbe molto piu' veloce ma la qualita' + * cala: -5040.33 -> -5158.68 di log-lik su 1023 token (~+12% perplexity). Il batch da + * solo e' bit-identical all'originale; il kernel no. Vedi issue. + * EN: batch the three projections over all S rows, like o_proj below and moe()'s + * batch-union. matmul_qt_ex(...,0) keeps them on the EXACT int4 kernel: letting S>1 pull + * them into IDOT is much faster but costs ~12% perplexity (measured). Batching alone is + * bit-identical to upstream; the kernel switch is not. */ + int pipe_done=0; +#ifdef COLI_CUDA + if(g_cuda_pipe&&!kvs&&S>=8&&layern_layers&&g_cuda_enabled&&c->kv_lora<=512&& + !(m->has_dsa&&pos_base+S>c->index_topk)&& + l->q_a.cuda_eligible&&l->q_b.cuda_eligible&&l->kv_a.cuda_eligible&& + l->kv_b.cuda_eligible&&l->o.cuda_eligible&& + qt_cuda_upload(&l->q_a)&&qt_cuda_upload(&l->q_b)&&qt_cuda_upload(&l->kv_a)&& + qt_cuda_upload(&l->kv_b)&&qt_cuda_upload(&l->o)) + pipe_done=attn_pipe_prefill(m,l,layer,x,0,S,pos_base,out,NULL); +#endif + if(!pipe_done){ + matmul_qt_ex(QR, x, &l->q_a, S, 0); + for(int s=0;sq_lora; + rmsnorm(qr, qr, l->q_a_ln, c->q_lora, c->eps); } /* q_b legge il residuo NORMATO */ + matmul_qt_ex(Q, QR, &l->q_b, S, 0); + matmul_qt_ex(comp, x, &l->kv_a, S, 0); + } + if(!pipe_done) for(int s=0;skv; - const float *xs=x+(int64_t)s*D; int pos=positions?positions[s]:pos_base+s; - float *qresid=QR+(int64_t)s*c->q_lora; - matmul_qt(qresid, xs, &l->q_a, 1); - rmsnorm(qresid, qresid, l->q_a_ln, c->q_lora, c->eps); - float *qfull=Q+(int64_t)s*H*qh; matmul_qt(qfull, qresid, &l->q_b, 1); + int pos=positions?positions[s]:pos_base+s; + float *qfull=Q+(int64_t)s*H*qh; for(int h=0;hqk_nope, pos, c); - matmul_qt(comp, xs, &l->kv_a, 1); + const float *cs=comp+(int64_t)s*cw; float *Ldst=coli_kv_row(ks->Lc[layer],pos,c->kv_lora); float *Rdst=coli_kv_row(ks->Rc[layer],pos,c->qk_rope); - memcpy(Ldst, comp, c->kv_lora*sizeof(float)); +#ifdef COLI_CUDA + if(ks==m->kv&&m->kv_dev_valid&&layer<=c->n_layers&&m->kv_dev_valid[layer]>pos) + m->kv_dev_valid[layer]=pos; /* riga riscritta: l'ombra si accorcia */ +#endif + memcpy(Ldst, cs, c->kv_lora*sizeof(float)); rmsnorm(Ldst, Ldst, l->kv_a_ln, c->kv_lora, c->eps); /* latente normato */ - memcpy(Rdst, comp+c->kv_lora, c->qk_rope*sizeof(float)); + memcpy(Rdst, cs+c->kv_lora, c->qk_rope*sizeof(float)); rope_interleave(Rdst, pos, c); /* k_rot roped, condiviso fra teste */ } /* ---- DSA lightning indexer ---- @@ -1605,14 +2424,23 @@ static void attention_rows(Model *m, Layer *l, int layer, float *x, int S, int p if(m->has_dsa && layern_layers && ((!kvs && m->kv_start[layer]==0) || kvs)){ int nh=c->index_nh, hd=c->index_hd; dtopk=c->index_topk; if(c->idx_type[layer]){ + /* BATCH-ROWS, come le proiezioni di attenzione sopra: ix_wk (D x index_hd) veniva + * ri-letto per OGNI token. matmul_qt_ex(...,0) lo tiene sul kernel int4 ESATTO: + * il batch da solo supererebbe il gate S>=g_i4s e cambierebbe la quantizzazione + * delle attivazioni. Cosi' l'output resta bit-identical. + * EN: batch ix_wk over all S rows like the attention projections; allow_idot=0 + * keeps it on the exact int4 kernel so the result stays bit-identical. */ + float *KD=falloc((int64_t)S*hd); + matmul_qt_ex(KD, x, &m->ix_wk[layer], S, 0); for(int s=0;skv; - const float *xs=x+(int64_t)s*D; int pos=positions?positions[s]:pos_base+s; + int pos=positions?positions[s]:pos_base+s; float *kd=coli_kv_row(ks->Ic[layer],pos,hd); - matmul_qt(kd, xs, &m->ix_wk[layer], 1); + memcpy(kd, KD+(int64_t)s*hd, (size_t)hd*sizeof(float)); layernorm(kd, m->ix_knw[layer], m->ix_knb[layer], hd, 1e-6f); rope_interleave(kd, pos, c); /* primi qk_rope dim, interleaved */ } + free(KD); if((int64_t)S*dtopk > m->dsa_scap){ free(m->dsa_sel); free(m->dsa_nsel); m->dsa_scap=(int64_t)S*dtopk; @@ -1659,7 +2487,17 @@ static void attention_rows(Model *m, Layer *l, int layer, float *x, int S, int p * k/v per ogni token del contesto. Per linearita': * q·k_nope_t = (W_K^hT q_nope)·L_t ctx^h = W_V^h (Σ_t a_t L_t) * costo per step ~O(T·kv_lora) invece di O(T·H·(nope+vh)) del matmul kvb_all. */ - int absorb = kvs || g_absorb==1 || (g_absorb<0 && S<=4); + if(pipe_done){ + free(ctx); free(Q); free(QR); free(comp); + m->t_attn += now_s()-ta0; + return; + } + int cuda_absorb=0; +#ifdef COLI_CUDA + cuda_absorb=layern_layers&&!kvs&&g_cuda_enabled&&getenv("COLI_CUDA_ATTN")&& + atoi(getenv("COLI_CUDA_ATTN"))&&c->kv_lora<=512; +#endif + int absorb = kvs || g_absorb==1 || (g_absorb<0 && S<=4) || cuda_absorb; if(absorb && c->kv_lora<=512){ m->t_aproj+=now_s()-ta0; double tac=now_s(); int kvl=c->kv_lora, r0v=c->qk_nope; /* offset righe V dentro il blocco di testa */ @@ -1678,19 +2516,48 @@ static void attention_rows(Model *m, Layer *l, int layer, float *x, int S, int p if(nt>sc_cap) sc_cap=nt; } float *sc_all = falloc((int64_t)omp_get_max_threads()*sc_cap); - int cuda_core=0; + int cuda_core=0,cuda_projected=0; #ifdef COLI_CUDA - if(S<=4&&g_cuda_enabled&&getenv("COLI_CUDA_ATTN")&&atoi(getenv("COLI_CUDA_ATTN"))&& + if(cuda_absorb&&l->n_kv_b_shard>1){ + int n=l->n_kv_b_shard,st0=m->kv_start[layer],nt=pos_base+S-st0,ok=1; + float *qs=falloc((int64_t)S*H*qh),*cs=falloc((int64_t)S*H*vh); + for(int d=0;dshard_h0[d]*S*qh+(int64_t)s*l->shard_hn[d]*qh, + Q+((int64_t)s*H+l->shard_h0[d])*qh,(size_t)l->shard_hn[d]*qh*sizeof(float)); + #pragma omp parallel for schedule(static) reduction(&:ok) + for(int d=0;dkv_b_shard[d], + cs+(int64_t)l->shard_h0[d]*S*vh,qs+(int64_t)l->shard_h0[d]*S*qh, + coli_kv_row(m->Lc[layer],st0,kvl),coli_kv_row(m->Rc[layer],st0,c->qk_rope), + S,l->shard_hn[d],c->qk_nope,c->qk_rope,vh,kvl,nt,c->attn_scale); + if(ok)for(int d=0;dshard_h0[d])*vh, + cs+(int64_t)l->shard_h0[d]*S*vh+(int64_t)s*l->shard_hn[d]*vh, + (size_t)l->shard_hn[d]*vh*sizeof(float)); + free(qs);free(cs);cuda_core=ok; + } else if(cuda_absorb&&l->kv_b.cuda_eligible&&l->o.cuda_eligible&& + qt_cuda_upload(&l->kv_b)&&qt_cuda_upload(&l->o)){ + int st0=m->kv_start[layer],nt=pos_base+S-st0; + cuda_core=cuda_projected=coli_cuda_attention_project_batch(l->kv_b.cuda,l->o.cuda,out,Q, + coli_kv_row(m->Lc[layer],st0,kvl),coli_kv_row(m->Rc[layer],st0,c->qk_rope), + S,H,c->qk_nope,c->qk_rope,vh,kvl,nt,c->attn_scale); + } else if(S<=4&&g_cuda_enabled&&getenv("COLI_CUDA_ATTN")&&atoi(getenv("COLI_CUDA_ATTN"))&& l->kv_b.cuda_eligible&&qt_cuda_upload(&l->kv_b)){ cuda_core=1; for(int s=0;skv;int pos=positions?positions[s]:pos_base+s; int st0=ks->kv_start[layer],nt=pos+1-st0; if(dnsel&&dnsel[s]>0){cuda_core=0;break;} - cuda_core=coli_cuda_attention_absorb(l->kv_b.cuda,ctx+(int64_t)s*H*vh, - Q+(int64_t)s*H*qh,coli_kv_row(ks->Lc[layer],st0,kvl), - coli_kv_row(ks->Rc[layer],st0,c->qk_rope),H,c->qk_nope,c->qk_rope, - vh,kvl,nt,c->attn_scale); + cuda_core=0; + if(g_cuda_pipe&&ks==m->kv&&layern_layers&&kv_dev_sync(m,l,layer,pos+1)) + cuda_core=coli_cuda_attention_absorb_kvdev(l->kv_b.cuda,ctx+(int64_t)s*H*vh, + Q+(int64_t)s*H*qh,m->kv_dev_L[layer]+(size_t)st0*kvl, + m->kv_dev_R[layer]+(size_t)st0*c->qk_rope,H,c->qk_nope,c->qk_rope, + vh,kvl,nt,c->attn_scale); + if(!cuda_core) + cuda_core=coli_cuda_attention_absorb(l->kv_b.cuda,ctx+(int64_t)s*H*vh, + Q+(int64_t)s*H*qh,coli_kv_row(ks->Lc[layer],st0,kvl), + coli_kv_row(ks->Rc[layer],st0,c->qk_rope),H,c->qk_nope,c->qk_rope, + vh,kvl,nt,c->attn_scale); } } #endif @@ -1725,7 +2592,7 @@ static void attention_rows(Model *m, Layer *l, int layer, float *x, int S, int p } } m->t_acore+=now_s()-tac; double tao=now_s(); - matmul_qt(out, ctx, &l->o, S); m->t_aout+=now_s()-tao; + if(!cuda_projected){matmul_qt(out, ctx, &l->o, S);} m->t_aout+=now_s()-tao; free(ctx); free(Q); free(QR); free(comp); free(sc_all); m->t_attn += now_s()-ta0; return; @@ -1779,26 +2646,45 @@ static void attention(Model *m, Layer *l, int layer, float *x, int S, int pos_ba * una volta sola e moltiplicato per tutte le posizioni che lo usano (pesi letti 1 volta); * lo shared expert e' un unico matmul a S righe. Per posizione l'accumulo resta * nell'ordine (routed nel loro ordine di union, poi shared). */ -static void moe(Model *m, Layer *l, int layer, float *x, int S, float *out){ +/* pin ∪ LRU residency probe (used by CACHE_ROUTE max-rank fill). */ +static int expert_is_resident(Model *m, int layer, int eid){ + ESlot *P=m->pin[layer]; + for(int z=0;znpin[layer];z++) if(P[z].eid==eid) return 1; + ESlot *Sl=m->ecache[layer]; + for(int z=0;zecn[layer];z++) if(Sl[z].eid==eid) return 1; + return 0; +} + +static void moe(Model *m, Layer *l, int layer, float *x, int S, float *out, int with_shared){ if(g_pilot_real){ /* barriera cross-layer: prendi possesso di QUESTO layer e aspetta * l'eventuale load-pilota in volo sullo stesso layer (dopodiche' il * worker droppa ogni nuovo load <= layer -> ecache[layer] e' stabile * per tutto il resolve/matmul/promozione qui sotto). */ - for(;;){ - pthread_mutex_lock(&g_pilot_mx); - atomic_store_explicit(&g_cur_moe_layer,layer,memory_order_release); - int inf=atomic_load_explicit(&g_pilot_inflight,memory_order_acquire); - pthread_mutex_unlock(&g_pilot_mx); - if(inf!=layer) break; - sched_yield(); - } + pthread_mutex_lock(&g_pilot_mx); + atomic_store_explicit(&g_cur_moe_layer,layer,memory_order_release); + while(layer>=0 && layer<256 && g_pilot_inflight[layer]>0) + pthread_cond_wait(&g_pilot_cv,&g_pilot_mx); + pthread_mutex_unlock(&g_pilot_mx); } Cfg *c=&m->c; int D=c->hidden, E=c->n_experts, K=c->topk, I=c->moe_inter; - float *logit=falloc(E), *choice=falloc(E); + float *choice=falloc(E); int sI=c->moe_inter*c->n_shared; + /* Rank buffer for CACHE_ROUTE max-rank selection (up to all E experts). */ + int *rank_buf=NULL; float *rank_w=NULL; + int do_cache_route = g_cache_route && E>0 && K>0; + int rank_cap = do_cache_route ? (g_route_m>K?g_route_m:K) : 0; + if(rank_cap>E) rank_cap=E; + if(do_cache_route){ + rank_buf=malloc((size_t)rank_cap*sizeof(int)); + rank_w=malloc((size_t)rank_cap*sizeof(float)); + if(!rank_buf||!rank_w){ free(rank_buf); free(rank_w); rank_buf=NULL; rank_w=NULL; do_cache_route=0; } + } /* ---- FASE A: routing di tutte le S posizioni ---- */ int *idxs=malloc((size_t)S*K*sizeof(int)); float *ws=malloc((size_t)S*K*sizeof(float)); int *keff=malloc(S*sizeof(int)); + /* router in UN matmul batch: stessa matematica, via le S chiamate S=1 */ + float *logits_all=falloc((int64_t)S*E); + int pre_routed=0; (void)pre_routed; #ifdef COLI_METAL if(g_pre_idx){ /* routing gia' calcolata dal layer CB (GPU) */ memcpy(idxs,g_pre_idx,(size_t)S*K*sizeof(int)); @@ -1813,18 +2699,111 @@ static void moe(Model *m, Layer *l, int layer, float *x, int S, float *out){ } for(int d=0;drouter, S, D, E); + if(!pre_routed) for(int s=0;srouter, 1, D, E); + float *logit=logits_all+(int64_t)s*E; for(int e=0;erouter_bias[e]; } int *idx=idxs+(int64_t)s*K; float *w=ws+(int64_t)s*K; int Ksel = g_topk>0 ? (g_topkbv){bv=choice[e];best=e;} } - idx[kk]=best; w[kk]=logit[best]; + if(do_cache_route){ + /* Full ranking of top rank_cap experts by choice (bias-augmented). */ + int Mwin=rank_cap; + if(g_route_p>0.f && g_route_p<1.f){ + /* Cumulative-mass variant: grow M until mass covers ROUTE_P. */ + int Mmax=g_route_m>Ksel*4?g_route_m:Ksel*4; if(Mmax>E) Mmax=E; if(Mmax>rank_cap) Mmax=rank_cap; + for(int kk=0;kkbv){bv=choice[e];best=e;} } + rank_buf[kk]=best; rank_w[kk]=logit[best]; + } + float tot=1e-20f; for(int kk=0;kk0?rank_w[kk]:0; + float cum=0; Mwin=Ksel; + for(int kk=0;kk0?rank_w[kk]:0; + if(cum>=g_route_p*tot){ Mwin=kk+1; break; } Mwin=kk+1; } + if(Mwinbv){bv=choice[e];best=e;} } + rank_buf[kk]=best; rank_w[kk]=logit[best]; + } + } + int J=g_route_j; if(J<0) J=0; if(J>Ksel) J=Ksel; + int chosen=0; + /* Always take true top-J (even if uncached). */ + for(int kk=0;kkroute_slots+=(uint64_t)Ksel; + for(int kk=0;kkroute_swaps++; + } + /* Pad if somehow short (shouldn't happen). */ + while(chosen0.f && g_route_alpha<1.f){ + for(int kk=0;kkroute_agree_hit+=(uint64_t)ov; + m->route_agree_tot+=(uint64_t)Ksel; + float tsum=1e-20f, csum=1e-20f; + for(int t=0;t0?rank_w[t]:0; + for(int kk=0;kk0?w[kk]:0; + double kl=0; + for(int t=0;t0?rank_w[t]:0)/tsum; + if(pt<=0) continue; + double pc=1e-12; + for(int kk=0;kk0?w[kk]:0)/csum; break; } + kl+=pt*log(pt/pc); + } + m->route_kl_sum+=kl; m->route_kl_n++; + } + } else { + for(int kk=0;kkbv){bv=choice[e];best=e;} } + idx[kk]=best; w[kk]=logit[best]; + } + if(g_route_agree){ + m->route_agree_hit+=(uint64_t)Ksel; + m->route_agree_tot+=(uint64_t)Ksel; + m->route_kl_sum+=0; m->route_kl_n++; + } } int Ke=Ksel; if(g_topp>0 && g_topp<1.f){ @@ -1849,6 +2828,7 @@ static void moe(Model *m, Layer *l, int layer, float *x, int S, float *out){ } for(int d=0;deroute[layer][z]==idxs[kk]){ la_hit[0]++; break; } la_tot[0]+=Ke; } - for(int kind=0;kind<2;kind++) if(la_val[kind][layer]){ /* [1]/[2] vs predizioni */ + for(int kind=0;kind<3;kind++) if(la_val[kind][layer]){ /* score all prediction kinds */ for(int kk=0;kk corrupted prefill hidden state -> wrong KV cache -> + * repetitive garbage decode. The budget is only safe token-by-token, where the + * prefill KV cache is already correct. (woolcoxm, #292.) */ + if(g_expert_budget>0 && S<=4 && nu>g_expert_budget){ + /* compute aggregate gate weight per unique expert */ + float *wsum=falloc(nu); for(int j=0;jpin[layer]; + for(int z=0;znpin[layer];z++) if(P[z].eid==eid){ found=1; break; } + if(!found){ ESlot *Sl=m->ecache[layer]; int nn=m->ecn[layer]; + for(int z=0;zbv){ bv=wsum[j]; best=j; } + if(best<0) break; keep[best]=1; nkeep++; + } + /* build a lookup: for each expert id, is it kept? (reuse seen[]) */ + memset(seen,0,(size_t)E); + for(int j=0;jnorm_topk && w>0){ + float sm=0; for(int kk=0;kkrouted_scale; + } + } + } + /* compact uniq[] to kept experts only */ + int nu2=0; + for(int j=0;jnpin[layer];z++) if(P[z].eid==eid){ m->hits++; use[j]=&P[z]; break; } if(!use[j]){ ESlot *Sl=m->ecache[layer]; int nn=m->ecn[layer]; for(int z=0;zhits++; Sl[z].used=(uint64_t)__atomic_add_fetch(&m->eclock,1,__ATOMIC_RELAXED); use[j]=&Sl[z]; break; } } - if(!use[j]){ qof[j]=nmiss; use[j]=&m->ws[nmiss]; missk[nmiss++]=j; m->miss++; } + if(!use[j]){ qof[j]=nmiss; use[j]=&m->ws[nmiss]; missk[nmiss++]=j; m->miss++; + if(g_disk_split){ if(m->ld_ctx==1) m->miss_draft++; else if(m->ld_ctx==2) m->miss_absorb++; } } } int metal_done=0; #ifdef COLI_METAL @@ -1965,7 +3015,7 @@ static void moe(Model *m, Layer *l, int layer, float *x, int S, float *out){ } else { double t0=now_s(); /* ORIGINALE: blocking parallel load */ #pragma omp parallel for schedule(dynamic,1) for(int q=0;qws[q],1); - m->t_edisk += now_s()-t0; } + m->t_ewait += now_s()-t0; } /* blocking: whole load stalls compute */ } /* I/O ASINCRONO: readahead (WILLNEED) del blocco SUCCESSIVO mentre calcoliamo * questo — il kernel legge in background, le pread dopo trovano cache calda */ @@ -1994,7 +3044,7 @@ static void moe(Model *m, Layer *l, int layer, float *x, int S, float *out){ * correct (and free) when a subset falls back to the CPU. */ if(g_pipe && nmiss){ double tw=now_s(); for(int q=0;qt_edisk += now_s()-tw; } + m->t_ewait += now_s()-tw; } /* blocking: drain stalled compute */ MB_BUILD(1, 0); /* missed experts, now loaded */ if(nbb>0){ double t0=now_s(); @@ -2018,7 +3068,7 @@ static void moe(Model *m, Layer *l, int layer, float *x, int S, float *out){ * Stays ABOVE the METAL skip: a subset that fell back to the CPU still needs its * slot drained here, and under METAL the block-level drain above already ran (this * spin is then a no-op). */ - if(g_pipe && qof[j]>=0){ double tw=now_s(); pipe_wait(qof[j]); m->t_edisk += now_s()-tw; } + if(g_pipe && qof[j]>=0){ double tw=now_s(); pipe_wait(qof[j]); m->t_ewait += now_s()-tw; } /* blocking */ #ifdef COLI_METAL /* skip the subsets already computed on GPU */ if(g_metal_enabled && ((is_miss[j] && !cpu_miss) || (!is_miss[j] && !cpu_res))) continue; @@ -2113,22 +3163,43 @@ static void moe(Model *m, Layer *l, int layer, float *x, int S, float *out){ ESlot tmp=*dst; *dst=m->ws[q]; m->ws[q]=tmp; dst->used=(uint64_t)__atomic_add_fetch(&m->eclock,1,__ATOMIC_RELAXED); } } } - /* ---- FASE E: shared expert, un matmul a S righe (skipped se fuso nel blocco GPU) ---- */ - float *sg=falloc((int64_t)S*sI), *su=falloc((int64_t)S*sI); + /* ---- FASE E: shared expert (PIPE2: gia' sul device; Metal CB: gia' sommata) ---- */ + if(!with_shared) goto shared_done; + { + float *sg=NULL,*su=NULL;int shared_cuda=0; #ifdef COLI_METAL if(g_pre_sh){ for(int64_t z=0;z<(int64_t)S*D;z++) out[z]+=g_pre_sh[z]; shared_on_gpu=1; } + if(shared_on_gpu) shared_cuda=2; /* gia' sommato in out: salta calcolo e add */ #endif - if(!shared_on_gpu){ +#ifdef COLI_CUDA + int shared_min=getenv("COLI_CUDA_SHARED_W4A16_MIN_ROWS")? + atoi(getenv("COLI_CUDA_SHARED_W4A16_MIN_ROWS")):32; + if(shared_min<16)shared_min=16; + if(shared_cuda==0&&S>=shared_min&&!l->shared_w4a16_failed&&!omp_in_parallel()&&g_cuda_enabled&& + l->sh_gate.fmt==2&&l->sh_up.fmt==2&&l->sh_down.fmt==2&& + getenv("COLI_CUDA_SHARED_W4A16")&&atoi(getenv("COLI_CUDA_SHARED_W4A16"))&& + qt_cuda_upload(&l->sh_gate)&&qt_cuda_upload(&l->sh_up)&&qt_cuda_upload(&l->sh_down)){ + shared_cuda=coli_cuda_shared_mlp_w4a16(l->sh_gate.cuda,l->sh_up.cuda, + l->sh_down.cuda,hh,x,S); + if(!shared_cuda)l->shared_w4a16_failed=1; + } +#endif + if(!shared_cuda){ + sg=falloc((int64_t)S*sI);su=falloc((int64_t)S*sI); matmul_qt(sg, x, &l->sh_gate, S); matmul_qt(su, x, &l->sh_up, S); for(int64_t z=0;z<(int64_t)S*sI;z++) sg[z]=siluf(sg[z])*su[z]; matmul_qt(hh, sg, &l->sh_down, S); - for(int64_t z=0;z<(int64_t)S*D;z++) out[z]+=hh[z]; } - free(logit); free(choice); free(idxs); free(ws); free(keff); free(uniq); - free(xg); free(gg); free(uu); free(hh); free(rows); free(rw); free(sg); free(su); + if(shared_cuda!=2) for(int64_t z=0;z<(int64_t)S*D;z++) out[z]+=hh[z]; + free(sg); free(su); + } +shared_done: + free(logits_all); free(choice); free(idxs); free(ws); free(keff); free(uniq); + free(xg); free(gg); free(uu); free(hh); free(rows); free(rw); #ifdef COLI_CUDA - free(group_x); free(group_y); free(group_row); free(group_weight); + free(group_x);free(group_y); + free(group_row); free(group_weight); #endif } @@ -2143,10 +3214,53 @@ static void dense_mlp(Layer *l, float *x, int S, int D, int I, float *out){ /* LOOKA: predice il top-K del router del layer `target` dallo stato h (residual stream), * usando la STESSA pipeline del routing vero (post_ln -> router -> sigmoid+bias, top-K). - * kind 0 = stesso layer saltando l'attention, kind 1 = layer successivo. */ + * kind 0 = stesso layer saltando l'attention + * kind 1 = layer successivo (PILOT: stale state, 75.8% recall) + * kind 2 = two-step: approximate L's shared expert output, add to state, THEN predict L+1. + * The shared expert is resident (part of dense model), so this adds 3 small matmuls + * but no disk I/O. The corrected state includes the dominant part of MoE(L) that the + * stale PILOT prediction is missing. */ static void la_predict(Model *m, int target, const float *h, int kind){ Cfg *c=&m->c; Layer *l=&m->L[target]; int D=c->hidden, E=c->n_experts, K=c->topk; float *nrm=falloc(D), *ch=falloc(E); + + if(kind==2){ + /* Two-step: h is L's post-attention state (pre-MoE). We want to predict L+1's + * routing. The real L+1 router sees h + MoE(L). We approximate MoE(L) by + * computing ONLY the shared expert (resident, no disk) on the post_ln-normalized + * state, then add it to h before running L+1's router. + * + * target = L+1, so the layer we need the shared expert from is L = target-1. */ + int src_layer = target - 1; + if(src_layer < 0 || src_layer >= c->n_layers || !m->L[src_layer].sparse + || c->n_shared <= 0 || c->moe_inter <= 0){ + la_val[2][target] = 0; free(nrm); free(ch); return; + } + Layer *sl = &m->L[src_layer]; + int sI = c->moe_inter * c->n_shared; + float *snrm = falloc(D), *sg = falloc(sI), *su = falloc(sI); + float *sout = falloc(D), *hc = falloc(D); + rmsnorm(snrm, h, sl->post_ln, D, c->eps); + matmul_qt(sg, snrm, &sl->sh_gate, 1); + matmul_qt(su, snrm, &sl->sh_up, 1); + for(int i=0;ish_down, 1); + for(int i=0;ipost_ln, D, c->eps); + free(snrm); free(sg); free(su); free(sout); free(hc); + matmul(ch, nrm, l->router, 1, D, E); + for(int e=0;erouter_bias[e]; + int *pred = la_pred[2][target]; + for(int kk=0;kkbv){bv=ch[e];best=e;} } + pred[kk]=best; } + la_val[2][target]=1; + free(nrm); free(ch); + return; + } + + /* Baseline kinds 0 and 1: pure router on the given state */ rmsnorm(nrm,h,l->post_ln,D,c->eps); matmul(ch,nrm,l->router,1,D,E); for(int e=0;erouter_bias[e]; @@ -2192,7 +3306,7 @@ static void pilot_realload(Model *m, int layer, int eid){ else { int lru=0; for(int z=1;zeid=-1; /* nascondi dagli scan-hint mentre carica */ - atomic_store_explicit(&g_pilot_inflight,layer,memory_order_release); + g_pilot_inflight[layer]++; pthread_mutex_unlock(&g_pilot_mx); int rc=expert_load(m,layer,eid,dst,0); /* pread VERO — fuori dal lock, sovrapposto al compute; fatal=0: un errore su una speculazione NON deve uccidere il server */ @@ -2205,18 +3319,96 @@ static void pilot_realload(Model *m, int layer, int eid){ } else { atomic_fetch_add_explicit(&g_pilot_drops,1,memory_order_relaxed); /* load fallito: slot resta nascosto (eid=-1), mai pubblicato */ } - atomic_store_explicit(&g_pilot_inflight,-1,memory_order_release); + g_pilot_inflight[layer]--; + pthread_cond_broadcast(&g_pilot_cv); pthread_mutex_unlock(&g_pilot_mx); if(rc!=0) /* mai swallow silenzioso: logga (una riga) e prosegui */ fprintf(stderr,"[PILOT] load speculativo abbandonato: layer %d expert %d (I/O error/short read) — nessun impatto sull'output\n",layer,eid); } +#ifdef __linux__ +typedef struct { int layer,eid,li; ESlot *dst; } PilotUringDone; +static void pilot_uring_batch(Model *m){ + PilotUringDone done[URING_LOAD_MAX]; int nd=0; + uring_batch_reset(&g_ub_pilot); + unsigned r=__atomic_load_n(&pilot_r,__ATOMIC_ACQUIRE); + unsigned w=__atomic_load_n(&pilot_w,__ATOMIC_ACQUIRE); + while(r!=w && nd=256){ atomic_fetch_add_explicit(&g_pilot_drops,1,memory_order_relaxed); continue; } + pthread_mutex_lock(&g_pilot_mx); + if(layer<=atomic_load_explicit(&g_cur_moe_layer,memory_order_acquire)){ + atomic_fetch_add_explicit(&g_pilot_drops,1,memory_order_relaxed); + pthread_mutex_unlock(&g_pilot_mx); continue; + } + int found=0; ESlot *P=m->pin[layer]; + for(int z=0;znpin[layer];z++) if(P[z].eid==eid){found=1;break;} + ESlot *Sl=m->ecache[layer]; int nn=m->ecn[layer]; + for(int z=0;zecap){ slot=nn; m->ecn[layer]=nn+1; } + else{ + slot=-1; + for(int z=0;zeid=-(eid+2); /* visible reservation; never considered resident/evictable */ + g_pilot_inflight[layer]++; + pthread_mutex_unlock(&g_pilot_mx); + + int li=uring_load_add(&g_ub_pilot,m,layer,eid,dst,0); + if(li<0){ + pthread_mutex_lock(&g_pilot_mx); dst->eid=-1; g_pilot_inflight[layer]--; + pthread_cond_broadcast(&g_pilot_cv); pthread_mutex_unlock(&g_pilot_mx); + atomic_fetch_add_explicit(&g_pilot_drops,1,memory_order_relaxed); continue; + } + done[nd++]=(PilotUringDone){layer,eid,li,dst}; + } + __atomic_store_n(&pilot_r,r,__ATOMIC_RELEASE); + if(!nd) return; + if(uring_submit_batch(&g_ub_pilot)<0){ + int err=errno; + for(int i=0;ili,0); + pthread_mutex_lock(&g_pilot_mx); + if(rc==0){ + d->dst->eid=d->eid; + d->dst->used=(uint64_t)__atomic_add_fetch(&m->eclock,1,__ATOMIC_RELAXED); + atomic_fetch_add_explicit(&g_pilot_loads,1,memory_order_relaxed); + }else{ + d->dst->eid=-1; + atomic_fetch_add_explicit(&g_pilot_drops,1,memory_order_relaxed); + } + g_pilot_inflight[d->layer]--; + pthread_cond_broadcast(&g_pilot_cv); + pthread_mutex_unlock(&g_pilot_mx); + if(rc) fprintf(stderr,"[PILOT/URING] load speculativo abbandonato: layer %d expert %d: %s\n", + d->layer,d->eid,strerror(g_ub_pilot.load[d->li].error)); + } +} +#endif static void *pilot_worker(void *arg){ (void)arg; for(;;){ unsigned r=__atomic_load_n(&pilot_r,__ATOMIC_ACQUIRE); unsigned w=__atomic_load_n(&pilot_w,__ATOMIC_ACQUIRE); if(r==w){ usleep(200); continue; } - if(g_pilot_real) pilot_realload(pilot_m, pilot_q[r&4095].l, pilot_q[r&4095].e); + if(g_pilot_real){ +#ifdef __linux__ + if(g_uring){ pilot_uring_batch(pilot_m); continue; } +#endif + pilot_realload(pilot_m, pilot_q[r&4095].l, pilot_q[r&4095].e); + } else expert_prefetch(pilot_m, pilot_q[r&4095].l, pilot_q[r&4095].e); __atomic_store_n(&pilot_r,r+1,__ATOMIC_RELEASE); } @@ -2280,7 +3472,8 @@ static void couple_prefetch(Model *m, int layer, const int *idx, int Ke){ ESlot *P=m->pin[lt]; for(int z=0;znpin[lt] && !found;z++) if(P[z].eid==best) found=1; ESlot *Sl=m->ecache[lt]; - for(int z=0;zecn[lt] && !found;z++) if(Sl[z].eid==best) found=1; + for(int z=0;zecn[lt] && !found;z++) + if(Sl[z].eid==best || Sl[z].eid==-(best+2)) found=1; pthread_mutex_unlock(&g_pilot_mx); if(!found){ unsigned w=__atomic_load_n(&pilot_w,__ATOMIC_RELAXED); @@ -2298,8 +3491,32 @@ static void pilot_prefetch(Model *m, int lnext, const float *x, int S){ int K = g_pilot_ktopk ? g_pilot_k : c->topk; if(!pilot_m){ pilot_m=m; pthread_t t; pthread_create(&t,NULL,pilot_worker,NULL); } float *nrm=falloc(D), *ch=falloc(E); + /* Two-step workspace (allocated once, reused across positions) */ + float *snrm=NULL, *sg=NULL, *su=NULL, *sout=NULL, *hc=NULL; + int src_layer = lnext - 1; + int sI = 0; + int can_two = g_pilot_two && src_layer>=0 && src_layern_layers + && m->L[src_layer].sparse && c->n_shared>0 && c->moe_inter>0; + if(can_two){ + sI = c->moe_inter * c->n_shared; + snrm=falloc(D); sg=falloc(sI); su=falloc(sI); sout=falloc(D); hc=falloc(D); + } for(int s=0;spost_ln, D, c->eps); + const float *xs = x+(int64_t)s*D; + if(can_two){ + /* Two-step: approximate MoE(src_layer) via shared expert only (resident, no disk), + * then run lnext's router on the corrected state. */ + Layer *sl = &m->L[src_layer]; + rmsnorm(snrm, xs, sl->post_ln, D, c->eps); + matmul_qt(sg, snrm, &sl->sh_gate, 1); + matmul_qt(su, snrm, &sl->sh_up, 1); + for(int i=0;ish_down, 1); + for(int i=0;ipost_ln, D, c->eps); + } else { + rmsnorm(nrm, xs, l->post_ln, D, c->eps); + } matmul(ch, nrm, l->router, 1, D, E); for(int e=0;erouter_bias[e]; for(int kk=0;kkpin[lnext]; for(int z=0;znpin[lnext] && !found;z++) if(P[z].eid==best) found=1; ESlot *Sl=m->ecache[lnext]; - for(int z=0;zecn[lnext] && !found;z++) if(Sl[z].eid==best) found=1; + for(int z=0;zecn[lnext] && !found;z++) + if(Sl[z].eid==best || Sl[z].eid==-(best+2)) found=1; pthread_mutex_unlock(&g_pilot_mx); if(!found){ unsigned w=__atomic_load_n(&pilot_w,__ATOMIC_RELAXED); @@ -2329,9 +3547,90 @@ static void pilot_prefetch(Model *m, int lnext, const float *x, int S){ } } free(nrm); free(ch); + if(can_two){ free(snrm); free(sg); free(su); free(sout); free(hc); } } /* forward di UN layer (usato dai 78 principali e dal layer MTP) */ +#ifdef COLI_CUDA +/* Inc.2a — intero layer SPARSO residente sul device del layer. x_dev entra e resta; + * lasciano il device solo: nrm post-attention (router + expert CPU + gather dei + * gruppi), i nuovi record KV, e la nrm pre-attention sui layer con indexer DSA. + * Ritorna 0 su errore: il chiamante ripristina lo snapshot e rifa' il layer su CPU. */ +static int pipe_layer_sparse(Model *m, Layer *l, int li, float *x_dev, int S, int pos_base, + float *nrm_host, float *out_host){ + Cfg *c=&m->c; int D=c->hidden, dev=l->kv_b.cuda_device; + int sI=c->moe_inter*c->n_shared; + size_t xb=(size_t)S*D*4; + if(!l->sh_gate.cuda_eligible||!l->sh_up.cuda_eligible||!l->sh_down.cuda_eligible|| + !qt_cuda_upload(&l->sh_gate)||!qt_cuda_upload(&l->sh_up)||!qt_cuda_upload(&l->sh_down)|| + l->sh_gate.cuda_device!=dev||l->sh_up.cuda_device!=dev||l->sh_down.cuda_device!=dev) return 0; + float *w_in =coli_cuda_pipe_scratch(dev,8,(size_t)D*4); + float *w_post=coli_cuda_pipe_scratch(dev,9,(size_t)D*4); + float *nrm_d=coli_cuda_pipe_scratch(dev,10,xb); + float *y_d =coli_cuda_pipe_scratch(dev,11,xb); + float *sg_d =coli_cuda_pipe_scratch(dev,12,(size_t)S*sI*4); + float *su_d =coli_cuda_pipe_scratch(dev,13,(size_t)S*sI*4); + float *snap =coli_cuda_pipe_scratch(dev,14,xb); + if(!w_in||!w_post||!nrm_d||!y_d||!sg_d||!su_d||!snap) return 0; + if(!coli_cuda_pipe_peer_copy(dev,snap,dev,x_dev,xb)) return 0; /* snapshot per il fallback */ + if(!coli_cuda_pipe_upload(dev,w_in,l->in_ln,(size_t)D*4)|| + !coli_cuda_pipe_upload(dev,w_post,l->post_ln,(size_t)D*4)) return 0; + double ta=now_s(); + if(!coli_cuda_pipe_rmsnorm(dev,nrm_d,x_dev,w_in,S,D,c->eps)) return 0; + /* DSA: i layer con indexer FULL cachano k_idx dalla nrm pre-attention (CPU, piccolo) */ + if(m->has_dsa && lin_layers && m->kv_start[li]==0 && c->idx_type[li]){ + if(!coli_cuda_pipe_download(dev,nrm_d,nrm_host,xb)) return 0; + int nh=c->index_nh, hd=c->index_hd; (void)nh; + for(int s=0;skv->Ic[li],pos,hd); + matmul_qt(kd, nrm_host+(int64_t)s*D, &m->ix_wk[li], 1); + layernorm(kd, m->ix_knw[li], m->ix_knb[li], hd, 1e-6f); + rope_interleave(kd, pos, c); + } + } + if(!attn_pipe_prefill(m,l,li,nrm_d,1,S,pos_base,NULL,y_d)) return 0; + if(!coli_cuda_pipe_add(dev,x_dev,y_d,(size_t)S*D)) return 0; /* prima mutazione */ + if(!coli_cuda_pipe_rmsnorm(dev,nrm_d,x_dev,w_post,S,D,c->eps)) return 0; + if(!coli_cuda_pipe_download(dev,nrm_d,nrm_host,xb)) return 0; + m->t_attn+=now_s()-ta; + /* OVERLAP: issue the shared expert on the GPU BEFORE moe() runs on the CPU. + * The shared expert reads nrm_d (valid after the download above) and writes its + * residual into x_dev (async). While the GPU computes this, the CPU enters moe() + * for routing + expert disk loads + matmul — ~50ms of work that previously left + * the GPU idle. The shared expert (~0.5ms) finishes early in that window. + * + * After moe(), the routed-expert result is uploaded (sync pipe_upload) and added + * to x_dev (async). Both residual adds (shared + routed) are ordered on the same + * stream — the next layer's pipe_rmsnorm reads x_dev after both complete. + * + * No pipe_sync at the end: the next layer's pipe_download (sync cudaMemcpy) + * provides the implicit sync point. The fallback path (caller downloads x_dev) + * also uses pipe_download which syncs. This lets GPU work chain across layers + * without a per-layer stall. + * + * Profiling: moe() self-times its own t_emm (routed expert matmul). We time only + * the GPU work that moe() does NOT cover: the shared-expert dispatch and the + * routed-expert upload+add. Previously a single outer span wrapped everything + * including moe(), double-counting the routed-expert time and driving the + * profile's "other" bucket negative (#292). */ + double te=now_s(); + if(!coli_cuda_pipe_gemm(l->sh_gate.cuda,sg_d,nrm_d,S)) return 0; + if(!coli_cuda_pipe_gemm(l->sh_up.cuda,su_d,nrm_d,S)) return 0; + if(!coli_cuda_pipe_silu_mul(dev,sg_d,su_d,(size_t)S*sI)) return 0; + if(!coli_cuda_pipe_gemm(l->sh_down.cuda,y_d,sg_d,S)) return 0; + if(!coli_cuda_pipe_add(dev,x_dev,y_d,(size_t)S*D)) return 0; /* shared residual (async) */ + m->t_emm += now_s()-te; /* shared-expert GPU dispatch only */ + /* expert routed su CPU/gruppi GPU come oggi (shared saltata: la fa il device) */ + moe(m,l,li,nrm_host,S,out_host,0); /* self-times its own t_emm */ + te=now_s(); + if(!coli_cuda_pipe_upload(dev,y_d,out_host,xb)) return 0; /* sync: waits for moe */ + if(!coli_cuda_pipe_add(dev,x_dev,y_d,(size_t)S*D)) return 0; /* routed residual (async) */ + m->t_emm += now_s()-te; /* routed-expert upload + add only */ + return 1; +} +#endif + static void layer_forward_rows(Model *m, Layer *l, int li, float *x, int S, int pos_base, KVState *const *kvs, const int *positions, float *nrm, float *tmp){ Cfg *c=&m->c; int D=c->hidden; @@ -2341,8 +3640,10 @@ static void layer_forward_rows(Model *m, Layer *l, int li, float *x, int S, int #ifdef COLI_METAL /* FULL-LAYER CB: in_ln + attention + residuo + post_ln + shared expert + router/top-K * in un solo submit GPU; la CPU legge il routing e fa solo resolve/disk/expert-CB. - * Fallback: qualsiasi condizione mancante -> percorso CPU intero qui sotto. */ - if(g_metal_enabled && S<=4 && lin_layers && l->sparse + * Fallback: qualsiasi condizione mancante -> percorso CPU intero qui sotto. + * !kvs: ragged mux rows (per-row KV/position) are not expressible in this kernel's + * single Lc/Rc + pos_base contract — see the matching guard in attention_rows. */ + if(g_metal_enabled && !kvs && S<=4 && lin_layers && l->sparse && (g_absorb==1||(g_absorb<0&&S<=4)) && m->kv_start[li]==0 && D==6144 && c->n_heads==64 && c->q_lora==2048 && c->kv_lora==512 && c->qk_nope==192 && c->qk_rope==64 && c->v_head==256 && l->kv_b.fmt==2 @@ -2382,9 +3683,12 @@ static void layer_forward_rows(Model *m, Layer *l, int li, float *x, int S, int } } if(g_pilot && S<=8 && li+1n_layers && m->L[li+1].sparse) pilot_prefetch(m,li+1,x,S); - if(g_looka && S==1 && li+1n_layers && m->L[li+1].sparse) la_predict(m,li+1,x,1); + if(g_looka && S==1 && li+1n_layers && m->L[li+1].sparse){ + la_predict(m,li+1,x,1); + la_predict(m,li+1,x,2); + } g_pre_idx=lidx; g_pre_w=lw; g_pre_keff=lkeff; g_pre_sh=lsh; - moe(m,l,li,lnrm,S,tmp); + moe(m,l,li,lnrm,S,tmp,1); g_pre_idx=NULL; g_pre_w=NULL; g_pre_keff=NULL; g_pre_sh=NULL; for(int64_t j=0;j<(int64_t)S*D;j++) x[j]+=tmp[j]; return; @@ -2396,9 +3700,12 @@ static void layer_forward_rows(Model *m, Layer *l, int li, float *x, int S, int attention_rows(m,l,li,nrm,S,pos_base,kvs,positions,tmp); for(int64_t j=0;j<(int64_t)S*D;j++) x[j]+=tmp[j]; if(g_pilot && S<=8 && li+1n_layers && m->L[li+1].sparse) pilot_prefetch(m,li+1,x,S); - if(g_looka && S==1 && li+1n_layers && m->L[li+1].sparse) la_predict(m,li+1,x,1); + if(g_looka && S==1 && li+1n_layers && m->L[li+1].sparse){ + la_predict(m,li+1,x,1); /* baseline: stale-state PILOT */ + la_predict(m,li+1,x,2); /* two-step: shared-expert-corrected prediction */ + } for(int s=0;spost_ln, D, c->eps); - if(l->sparse) moe(m,l,li,nrm,S,tmp); else dense_mlp(l,nrm,S,D,c->dense_inter,tmp); + if(l->sparse) moe(m,l,li,nrm,S,tmp,1); else dense_mlp(l,nrm,S,D,c->dense_inter,tmp); for(int64_t j=0;j<(int64_t)S*D;j++) x[j]+=tmp[j]; } static void layer_forward(Model *m, Layer *l, int li, float *x, int S, int pos_base, float *nrm, float *tmp){ @@ -2413,13 +3720,63 @@ static void layers_forward_rows(Model *m, float *x, int S, int pos_base, pthread_mutex_unlock(&g_pilot_mx); } float *nrm=falloc((int64_t)S*D), *tmp=falloc((int64_t)S*D); +#ifdef COLI_CUDA + /* PIPE2 (Inc.2a): il residuo resta sul device del layer, saltando tra le schede + * ai confini di layer. x host diventa STALE finche' la residenza e' attiva. + * + * S threshold is device-count-dependent (#273): on a single GPU the resident + * stream wins at S=1 (evicts the CPU round-trips that dominate small-batch + * decode — +49% on a 5070 Ti). With layers sharded across multiple GPUs each + * resident forward crosses P2P per layer group, and at one token per forward + * those hops don't amortize — A/B on 6x5090 showed S=1 is a wash there. So: + * single-GPU engages at S=1, multi-GPU keeps the original S>=8 prefill gate. + * COLI_CUDA_PIPE_S_MIN overrides for anyone who wants to measure. */ + float *x_dev=NULL; int x_dev_on=-1; + size_t xb=(size_t)S*(size_t)D*4; + int pipe_s_min = getenv("COLI_CUDA_PIPE_S_MIN") ? atoi(getenv("COLI_CUDA_PIPE_S_MIN")) + : (g_cuda_ndev<=1 ? 1 : 8); + int pipe2 = g_cuda_pipe>=2 && !kvs && S>=pipe_s_min && g_cuda_enabled && c->kv_lora<=512 && + !(m->has_dsa && pos_base+S>c->index_topk); +#endif for(int i=0;in_layers;i++){ /* progresso su stderr per i batch grossi (prefill): il primo byte di risposta * puo' arrivare dopo MINUTI di streaming — al buio sembra un blocco. */ if(S>=8 && (i%4==0 || i==c->n_layers-1)) fprintf(stderr,"[prefill] layer %d/%d · %d token\n", i+1, c->n_layers, S); +#ifdef COLI_CUDA + Layer *l=&m->L[i]; + if(pipe2 && l->sparse && in_layers && + l->q_a.cuda_eligible&&l->q_b.cuda_eligible&&l->kv_a.cuda_eligible&& + l->kv_b.cuda_eligible&&l->o.cuda_eligible&& + qt_cuda_upload(&l->q_a)&&qt_cuda_upload(&l->q_b)&&qt_cuda_upload(&l->kv_a)&& + qt_cuda_upload(&l->kv_b)&&qt_cuda_upload(&l->o)&& + l->q_a.cuda_device==l->kv_b.cuda_device&&l->q_b.cuda_device==l->kv_b.cuda_device&& + l->kv_a.cuda_device==l->kv_b.cuda_device&&l->o.cuda_device==l->kv_b.cuda_device){ + int dev=l->kv_b.cuda_device, ok=1; + float *dst=coli_cuda_pipe_scratch(dev,15,xb); + if(dst){ + if(x_dev_on<0) ok=coli_cuda_pipe_upload(dev,dst,x,xb); + else if(x_dev_on!=dev){ ok=coli_cuda_pipe_peer_copy(dev,dst,x_dev_on,x_dev,xb); } + else dst=x_dev; + if(ok){ + x_dev=dst; x_dev_on=dev; + if(pipe_layer_sparse(m,l,i,x_dev,S,pos_base,nrm,tmp)) continue; + /* fallback: snapshot -> host, layer rifatto sul percorso CPU */ + coli_cuda_pipe_peer_copy(dev,x_dev,dev,coli_cuda_pipe_scratch(dev,14,xb),xb); + coli_cuda_pipe_download(dev,x_dev,x,xb); + x_dev_on=-1; + }else x_dev_on=-1; + } + } else if(x_dev_on>=0){ /* layer fuori pipe: il residuo torna a casa */ + coli_cuda_pipe_download(x_dev_on,x_dev,x,xb); + x_dev_on=-1; + } +#endif layer_forward_rows(m,&m->L[i],i,x,S,pos_base,kvs,positions,nrm,tmp); } +#ifdef COLI_CUDA + if(x_dev_on>=0) coli_cuda_pipe_download(x_dev_on,x_dev,x,xb); +#endif free(nrm); free(tmp); } static void layers_forward(Model *m, float *x, int S, int pos_base){ @@ -2429,6 +3786,13 @@ static void layers_forward(Model *m, float *x, int S, int pos_base){ static void kv_alloc(Model *m, int max_t){ Cfg *c=&m->c; KVState *k=m->kv; +#ifdef COLI_CUDA + if(m->kv_dev_L) for(int i=0;in_layers+1;i++){ /* dimensioni cambiate: ombra da rifare */ + if(m->kv_dev_L[i]){ coli_cuda_pipe_free(m->L[in_layers?i:0].kv_b.cuda_device,m->kv_dev_L[i]); m->kv_dev_L[i]=NULL; } + if(m->kv_dev_R[i]){ coli_cuda_pipe_free(m->L[in_layers?i:0].kv_b.cuda_device,m->kv_dev_R[i]); m->kv_dev_R[i]=NULL; } + m->kv_dev_valid[i]=0; + } +#endif if(k->Lc){ for(int i=0;in_layers+1;i++){ #ifdef COLI_METAL if(g_metal_enabled){ coli_metal_unregister(k->Lc[i]); coli_metal_unregister(k->Rc[i]); } @@ -2461,6 +3825,8 @@ static void kv_alloc(Model *m, int max_t){ } static void kv_bind(Model *m, KVState *k){ + if(m->kv!=k && m->kv_dev_valid) /* ombra legata al KVState corrente */ + for(int i=0;ic.n_layers+1;i++) m->kv_dev_valid[i]=0; m->kv=k; m->Lc=k->Lc; m->Rc=k->Rc; m->Ic=k->Ic; m->max_t=k->max_t; m->kv_start=k->kv_start; } @@ -2560,6 +3926,7 @@ static int mtp_draft(Model *m, int next_tok, int kv, int G, int *draft){ float *row=falloc(D), *logit=falloc(c->vocab), *h=falloc(D); memcpy(h, m->hlast, D*sizeof(float)); int tok=next_tok, n=0; + m->ld_ctx=1; /* DISK_SPLIT: i load da qui sono draft-path */ int prenorm = getenv("MTP_PRENORM")!=NULL; for(int g=0; g=m->max_t) break; @@ -2584,6 +3951,7 @@ static int mtp_draft(Model *m, int next_tok, int kv, int G, int *draft){ pos, tok, sqrt(n_eh), sqrt(n_post), t_pre, t2); draft[n++]=t2; tok=t2; memcpy(h, hx, D*sizeof(float)); } + m->ld_ctx=0; free(x); free(cat); free(hx); free(nrm); free(tmp); free(row); free(logit); free(h); return n; } @@ -2607,7 +3975,9 @@ static void mtp_absorb(Model *m, const int *next_ids, const float *x, int S, int matmul_qt(hx+(int64_t)i*D, cat, &m->eh_proj, 1); } float *nrm=falloc((int64_t)S*D), *tmp=falloc((int64_t)S*D); + m->ld_ctx=2; /* DISK_SPLIT: load del layer MTP in absorb */ layer_forward(m,&m->mtpL,li,hx,S,pos_base,nrm,tmp); + m->ld_ctx=0; free(hx); free(cat); free(e); free(hn); free(hf); free(nrm); free(tmp); } @@ -2621,23 +3991,36 @@ static inline int argmax_v(const float *lo, int V){ * gia' tokenizzato. Il confine di tokenizzazione non e' garantito coincidere con quello * del modello: la verifica assorbe la differenza (al peggio l'ultimo draft e' rifiutato). */ static void grammar_setup(Tok *T){ - const char *gf=getenv("GRAMMAR"); if(!gf||!*gf) return; - FILE *f=fopen(gf,"rb"); - if(!f){ fprintf(stderr,"[GRAMMAR] cannot open %s\n",gf); return; } + /* GRAMMAR= takes precedence; SCHEMA= compiles a JSON-Schema + * to GBNF (schema_gbnf.h) for the same draft source. Both fail soft: the engine + * runs without a grammar and output is unchanged. */ + const char *gf=getenv("GRAMMAR"); + const char *sf=(gf&&*gf)?NULL:getenv("SCHEMA"); + if((!gf||!*gf)&&(!sf||!*sf)) return; + const char *path=(gf&&*gf)?gf:sf; + FILE *f=fopen(path,"rb"); + if(!f){ fprintf(stderr,"[GRAMMAR] cannot open %s\n",path); return; } fseek(f,0,SEEK_END); long n=ftell(f); fseek(f,0,SEEK_SET); char *txt=malloc((size_t)n+1); if(!txt || fread(txt,1,(size_t)n,f)!=(size_t)n){ - fprintf(stderr,"[GRAMMAR] failed to read %s\n",gf); fclose(f); free(txt); return; } + fprintf(stderr,"[GRAMMAR] failed to read %s\n",path); fclose(f); free(txt); return; } fclose(f); txt[n]=0; - if(gr_parse(&g_gram,txt)){ fprintf(stderr,"[GRAMMAR] %s: %s\n",gf,g_gram.err); free(txt); return; } + if(sf){ /* schema -> GBNF, then the same gr_parse as the GRAMMAR path */ + char serr[160]; + char *gbnf=schema_to_gbnf(txt,serr,sizeof serr); + free(txt); + if(!gbnf){ fprintf(stderr,"[SCHEMA] %s: %s (running without grammar)\n",sf,serr); return; } + txt=gbnf; + } + if(gr_parse(&g_gram,txt)){ fprintf(stderr,"[GRAMMAR] %s: %s\n",path,g_gram.err); free(txt); return; } free(txt); gr_state_init(&g_gst,&g_gram); - if(!g_gst.alive){ fprintf(stderr,"[GRAMMAR] %s: grammar cannot be evaluated (left recursion?)\n",gf); return; } + if(!g_gst.alive){ fprintf(stderr,"[GRAMMAR] %s: grammar cannot be evaluated (left recursion?)\n",path); return; } if(getenv("GRAMMAR_DRAFT")) g_gr_max=atoi(getenv("GRAMMAR_DRAFT")); if(g_gr_max<1) g_gr_max=1; if(g_gr_max>48) g_gr_max=48; g_gr_T=T; g_gr_on=1; - fprintf(stderr,"[GRAMMAR] %s: %d rules, forced span capped at %d tokens/forward\n",gf,g_gram.n,g_gr_max); + fprintf(stderr,"[GRAMMAR] %s: %d rules, forced span capped at %d tokens/forward\n",path,g_gram.n,g_gr_max); } /* stato pulito all'inizio di ogni RISPOSTA (non tra i \x02MORE, che continuano) */ static void grammar_reset(void){ @@ -2725,6 +4108,8 @@ static int pick_tok(const float *lo, int V, int ban){ /* stop-set attivo (popolato da run_text/run_serve dal config; vuoto in validazione, * dove si genera un numero fisso di token da confrontare con l'oracolo) */ static int g_stop[9], g_nstop=0; +static void repin_pass_limit(Model *m,int limit); +static void repin_pass(Model *m){ repin_pass_limit(m,16); } static inline int is_stop(int t){ for(int i=0;i= kv+n_new+g_draft+2), kv = token gia' in KV. * logit = logits della posizione kv-1 (dal prefill); viene liberato qui. * emit(tok,ud) per ogni token emesso. Ritorna i token emessi; *kv_out = nuova kv. */ +/* STOP MORBIDO (serve/chat): SIGINT chiude il turno CORRENTE per la stessa via + * del tetto NGEN (stats, usage_save, KV append, sentinella END tutti normali) + * invece di uccidere il motore; :more puo' continuare la risposta interrotta. + * Il flag e' armato solo nei serve-loop (intr_install): nei run one-shot e in + * validazione SIGINT resta il default (morte immediata). Solo POSIX: su + * Windows il comportamento di Ctrl-C non cambia. + * EN: soft stop (serve/chat): SIGINT ends the CURRENT turn through the same + * path as the NGEN cap — stats/usage/KV/END sentinel all normal — instead of + * killing the engine; :more can continue the interrupted answer. Armed only + * in the serve loops; one-shot runs keep default SIGINT. POSIX only. */ +static volatile sig_atomic_t g_intr=0; +#if defined(__APPLE__) || defined(__linux__) || defined(__FreeBSD__) +static void intr_sig(int s){ (void)s; g_intr=1; } +static void intr_install(void){ + struct sigaction sa; memset(&sa,0,sizeof(sa)); + sa.sa_handler=intr_sig; sigemptyset(&sa.sa_mask); + sa.sa_flags=SA_RESTART; /* getline/pread non devono vedere EINTR */ + sigaction(SIGINT,&sa,NULL); +} +#else +static void intr_install(void){} +#endif static int spec_decode(Model *m, int *all, int kv, int n_new, int eos, float *logit, void (*emit)(int,void*), void *ud, int *kv_out){ Cfg *c=&m->c; int V=c->vocab; int emitted=0, done=0; int draft[64]; if(g_draft>63) g_draft=63; int carry_ban=-1; /* token rifiutato dalla verifica: escluso dal resample */ - while(emitted pin della famiglia di kernel per draft+verifica. + * EN: model drafts live -> pin the kernel family for draft+verify forwards. */ + g_spec_live = (g_draft>0); + if(spec_pinned() && m->has_mtp){ static int once=0; if(!once){ once=1; + fprintf(stderr,"[SPEC_PIN] draft+verify pinned to the S=1 kernel family: int4=%s int8=%s (#163; SPEC_PIN=0 for A/B)\n", + (g_idot&&g_i4s<=1)?"idot":"exact", g_idot?"idot":"exact"); } } + /* guardia MTP morbida (#163): finestra di 24 proposte, pausa e ri-arma invece del + * latch permanente — una regressione transitoria non spegne MTP per tutta la sessione. + * EN: soft MTP guard (#163): 24-proposal window, pause and re-arm instead of the + * permanent latch — a transient collapse no longer kills MTP for the whole session. */ + enum { GUARD_PAUSE_TOKENS = 256 }; + uint64_t gd_prop0=m->mtp_prop, gd_acc0=m->mtp_acc; int gd_pause=0; + while(emitted=0 && next==eos) || is_stop(next)) break; emit(next,ud); all[kv]=next; emitted++; m->n_emit++; @@ -2758,15 +4177,19 @@ static int spec_decode(Model *m, int *all, int kv, int n_new, int eos, float *lo g=grammar_draft(draft,g_gr_max); if(g>0) gsrc=1; } - if(!g && g_draft>0){ - /* auto-off adattivo: draft che non vengono mai accettati = solo tassa disco */ - if(m->has_mtp && m->mtp_prop>=24 && m->mtp_acc*10 < m->mtp_prop){ - g_draft=0; - fprintf(stderr,"[MTP] %.0f%% acceptance after %llu proposals: drafts disabled\n", - 100.0*m->mtp_acc/m->mtp_prop, (unsigned long long)m->mtp_prop); + if(!g && g_draft>0 && m->has_mtp){ + /* pausa adattiva: draft che non vengono mai accettati = solo tassa disco, + * ma il vecchio g_draft=0 era permanente. EN: adaptive pause; the old + * g_draft=0 latch was permanent. */ + if(gd_pause>0){ gd_pause--; if(!gd_pause){ gd_prop0=m->mtp_prop; gd_acc0=m->mtp_acc; } } + else if(m->mtp_prop-gd_prop0>=24 && (m->mtp_acc-gd_acc0)*10 < m->mtp_prop-gd_prop0){ + fprintf(stderr,"[MTP] %.0f%% acceptance over the last %llu proposals: drafts paused for %d tokens\n", + 100.0*(m->mtp_acc-gd_acc0)/(m->mtp_prop-gd_prop0), + (unsigned long long)(m->mtp_prop-gd_prop0), (int)GUARD_PAUSE_TOKENS); + gd_pause=GUARD_PAUSE_TOKENS; } } - if(!g && g_draft>0){ + if(!g && g_draft>0 && !(m->has_mtp && gd_pause>0)){ if(m->has_mtp){ g=mtp_draft(m,next,kv,g_draft,draft); m->mtp_prop+=g; if(g)gsrc=2; } else { g=ngram_draft(all,kv+1,g_draft,draft); if(g)gsrc=2; } } @@ -2796,7 +4219,9 @@ static int spec_decode(Model *m, int *all, int kv, int n_new, int eos, float *lo if(m->h_all && khlast, m->h_all+(int64_t)k*m->c.hidden, m->c.hidden*sizeof(float)); kv += 1+k; /* KV oltre kv e' stantia: verra' sovrascritta */ logit=falloc(V); memcpy(logit, lo+(int64_t)k*V, V*sizeof(float)); free(lo); + repin_pass(m); /* safe point: all device work is synchronized */ } + g_spec_live = 0; /* prefill/decode successivi: gate normali / next prefill: normal gates */ if(logit) free(logit); if(kv_out) *kv_out=kv; return emitted; @@ -2811,9 +4236,17 @@ static void emit_stream(int t, void *ud){ EmitStream *e=(EmitStream*)ud; char dec[64]; int dn=tok_decode(e->T,&t,1,dec,63); dec[dn]=0; fputs(dec,stdout); fflush(stdout); if(!e->quiet && ++e->count%16==0){ double tt=e->m->hits+e->m->miss; - fprintf(stderr,"\n[t=%d RSS %.2f GB hit %.0f%% %.2f tok/s %.2f tok/fw]\n", e->count, - rss_gb(), tt?100.0*e->m->hits/tt:0.0, e->count/(now_s()-e->t0), - e->m->n_fw?(double)e->m->n_emit/e->m->n_fw:1.0); } + if(g_cache_route && e->m->route_slots){ + double swap=100.0*e->m->route_swaps/e->m->route_slots; + fprintf(stderr,"\n[t=%d RSS %.2f GB hit %.0f%% swap %.0f%% %.2f tok/s %.2f tok/fw]\n", e->count, + rss_gb(), tt?100.0*e->m->hits/tt:0.0, swap, e->count/(now_s()-e->t0), + e->m->n_fw?(double)e->m->n_emit/e->m->n_fw:1.0); + } else { + fprintf(stderr,"\n[t=%d RSS %.2f GB hit %.0f%% %.2f tok/s %.2f tok/fw]\n", e->count, + rss_gb(), tt?100.0*e->m->hits/tt:0.0, e->count/(now_s()-e->t0), + e->m->n_fw?(double)e->m->n_emit/e->m->n_fw:1.0); + } + } } /* teacher-forcing: un solo forward su ids[S], argmax per posizione in pred[S] */ @@ -2826,7 +4259,7 @@ static void forward_all(Model *m, const int *ids, int S, int *pred){ float *lo=falloc(c->vocab); float *row=falloc(D); for(int s=0;sfinal_norm, D, c->eps); + rmsnorm(row, x+(int64_t)s*D, m->final_norm, D, c->eps); /* heap row (#183) */ matmul_qt(lo, row, &m->lm_head, 1); int best=0; float bv=lo[0]; for(int i=1;ivocab;i++) if(lo[i]>bv){bv=lo[i];best=i;} pred[s]=best; @@ -2841,16 +4274,43 @@ static double logprob_target(const float *lo, int V, int target, int *am){ if(am)*am=(best==target); return (double)(lo[target]-mx) - log(se); } +/* "glm" contenuto in model_type, case-insensitive — stessa regola di detect_prefix + * in tools/eval_glm.py (#194). Niente strcasestr: non esiste su MinGW/MSVC. */ +static int mt_is_glm(const char *s){ + if(s) for(;*s;s++) if((s[0]|32)=='g'&&(s[1]|32)=='l'&&(s[2]|32)=='m') return 1; + return 0; +} /* modalita' SCORING per i benchmark (stile lm-eval, log-likelihood): * input: file con righe " .. " (T=ctxlen+contlen) * output: riga " " per richiesta. * Un solo forward per richiesta (teacher-forcing): niente generazione -> fattibile a bassa velocita'. */ -static void run_score(Model *m, const char *path){ +static void run_score(Model *m, const char *snap, const char *path){ Cfg *c=&m->c; int D=c->hidden; + /* prefisso GLM (#108): il modello vede [gMASK] in testa a OGNI sequenza di training — + * scorare stream nudi e' out-of-distribution e deprime/distorce i punteggi. Se il config + * dice glm* gli id dei due token vengono chiesti al tokenizer.json dello snapshot (per + * GLM-5.2: 154822,154824 — mai fidarsi di costanti cablate, il vocabolario cambia tra + * release) e anteposti al CONTESTO delle richieste che non li hanno gia'; chi arriva GIA' + * prefissato (eval_glm.py post-#194) passa INTATTO. SCORE_PREFIX=0 -> comportamento nudo. */ + int pfx[2]={-1,-1}, pfx_on=0; + if(!getenv("SCORE_PREFIX")||atoi(getenv("SCORE_PREFIX"))){ + char *ar=NULL; jval *r=cfg_root(snap,&ar); + jval *mt=json_get(r,"model_type"); + if(mt_is_glm(mt?mt->str:NULL)){ + char tkp[2048]; snprintf(tkp,sizeof(tkp),"%s/tokenizer.json",snap); + Tok T; tok_load(&T,tkp); + pfx[0]=tok_id_of(&T,"[gMASK]"); pfx[1]=tok_id_of(&T,""); + if(pfx[0]>=0&&pfx[1]>=0){ pfx_on=1; + fprintf(stderr,"[SCORE] GLM snapshot: prepending [gMASK] (ids %d,%d) to unprefixed requests — disable with SCORE_PREFIX=0\n",pfx[0],pfx[1]); + } else fprintf(stderr,"[SCORE] GLM config but tokenizer has no [gMASK]/: prefix OFF\n"); + } + free(ar); + } FILE *f=fopen(path,"rb"); if(!f){perror(path);exit(1);} int maxT=1; { char *ln=NULL; size_t cp=0; while(getline(&ln,&cp,f)>0){ int a,b; if(sscanf(ln,"%d %d",&a,&b)==2 && a+b>maxT) maxT=a+b; } free(ln); } + if(pfx_on) maxT+=2; /* le richieste senza prefisso crescono di 2 token */ kv_alloc(m,maxT); float *x=falloc((int64_t)maxT*D), *lo=falloc(c->vocab), *row=falloc(D); int *ids=malloc(maxT*sizeof(int)); @@ -2859,6 +4319,10 @@ static void run_score(Model *m, const char *path){ char *p=ln; int ctxlen=strtol(p,&p,10), contlen=strtol(p,&p,10), T=ctxlen+contlen; if(T<=0||ctxlen<1){ printf("0 0 0\n"); fflush(stdout); continue; } for(int i=0;i=2 && ids[0]==pfx[0] && ids[1]==pfx[1])){ /* gia' prefissato -> intatto */ + memmove(ids+2,ids,(size_t)T*sizeof(int)); + ids[0]=pfx[0]; ids[1]=pfx[1]; ctxlen+=2; T+=2; + } for(int s=0;st_ewait+m->t_emm+m->t_attn+m->t_head; + double accounted=m->t_edisk+m->t_ewait+m->t_emm+m->t_attn+m->t_head; printf("PROFILE: expert-disk %.3fs service / %.3fs wait | expert-matmul %.3fs | attention %.3fs " "(including kvb %.3fs) | lm_head %.3fs | other %.3fs\n", m->t_edisk,m->t_ewait,m->t_emm,m->t_attn,m->t_kvb,m->t_head,elapsed-accounted); @@ -2900,6 +4364,11 @@ static void profile_print(Model *m, double elapsed){ #endif } +static void profile_reset(Model *m){ + m->t_edisk=m->t_ewait=m->t_emm=m->t_attn=m->t_kvb=m->t_head=0; + m->t_aproj=m->t_acore=m->t_aout=0; +} + /* Fixed-token decode benchmark: prefill all but the prompt's last token, then * replay the oracle sequence one token at a time. CPU and CUDA therefore see * identical hidden-state inputs even if their argmax predictions differ. */ @@ -2908,8 +4377,7 @@ static void run_replay(Model *m, const int *full, int nfull, int np){ kv_alloc(m,nfull+2); float *logit=step(m,full,np-1,0); free(logit); m->hits=m->miss=m->ereq=m->gpu_expert_calls=0; - m->t_edisk=m->t_ewait=m->t_emm=m->t_attn=m->t_kvb=m->t_head=0; - m->t_aproj=m->t_acore=m->t_aout=0; + profile_reset(m); double t0=now_s(); int steps=0; for(int i=np-1;i0){ + m->n_emit=(uint64_t)g_repin; + int limit=32; +#ifdef COLI_CUDA + if(m->gpu_expert_count) limit=m->c.n_layers; +#endif + repin_pass_limit(m,limit); /* prompt routing seeds every GPU layer */ + } + prefill_t=now_s()-prefill_t; + printf("PROFILO PREFILL (%.2fs):\n",prefill_t); profile_print(m,prefill_t); + m->hits=m->miss=m->ereq=m->gpu_expert_calls=0; + m->n_emit=m->n_fw=0; + g_last_repin=0; + profile_reset(m); + double t=now_s(); EmitStream es={&T,m,t,0,0}; grammar_reset(); int produced=spec_decode(m,all,np,ngen,eos,logit,emit_stream,&es,NULL); double dt=now_s()-t; double tot=m->hits+m->miss; int nsp=0; for(int i=0;in_layers;i++) if(m->L[i].sparse) nsp++; - printf("\n---\n%d tokens in %.2fs (%.2f tok/s) | expert hit rate %.1f%% | RSS %.2f GB\n", - produced, dt, produced/dt, tot?100.0*m->hits/tot:0.0, rss_gb()); - printf("experts loaded/token: %.1f (per-layer %.2f across %d; baseline topk=%d) | TOPK=%d TOPP=%.2f\n", + printf("\n---\nprefill %d tokens in %.2fs | decode %d tokens in %.2fs (%.2f tok/s) | " + "expert hit rate %.1f%% | RSS %.2f GB", + np,prefill_t,produced,dt,produced/dt,tot?100.0*m->hits/tot:0.0,rss_gb()); + if(g_cache_route && m->route_slots) + printf(" | swap %.1f%% (%llu/%llu)", + 100.0*m->route_swaps/m->route_slots, + (unsigned long long)m->route_swaps,(unsigned long long)m->route_slots); + if(m->route_agree_tot) + printf(" | route_agree %.1f%% | route_kl %.4f", + 100.0*m->route_agree_hit/m->route_agree_tot, + m->route_kl_n?m->route_kl_sum/(double)m->route_kl_n:0.0); + printf("\n"); + printf("experts loaded/token: %.1f (per-layer %.2f across %d; baseline topk=%d) | TOPK=%d TOPP=%.2f", produced?(double)m->ereq/produced:0.0, (produced&&nsp)?(double)m->ereq/produced/nsp:0.0, nsp, c->topk, g_topk, g_topp); + if(g_cache_route) printf(" | CACHE_ROUTE J=%d M=%d P=%.2f alpha=%.2f", g_route_j, g_route_m, g_route_p, g_route_alpha); + if(g_expert_budget) printf(" | EXPERT_BUDGET=%d (dropped %lld experts, ~%.1f GB I/O saved)", g_expert_budget, (long long)g_budget_dropped, g_budget_dropped*18.9e6/1e9); + printf("\n"); printf("speculation: %.2f tokens/forward (%llu forwards per %llu tokens) | MTP acceptance %.0f%% (%llu/%llu)\n", m->n_fw?(double)m->n_emit/m->n_fw:1.0, (unsigned long long)m->n_fw, (unsigned long long)m->n_emit, m->mtp_prop?100.0*m->mtp_acc/m->mtp_prop:0.0, (unsigned long long)m->mtp_acc, (unsigned long long)m->mtp_prop); if(g_cp_enq) printf("couple: %ld cross-layer prefetch hints enqueued\n", g_cp_enq); if(g_gr_prop) printf("grammar: %.0f%% acceptance (%llu/%llu forced drafts)\n", 100.0*g_gr_acc/g_gr_prop, (unsigned long long)g_gr_acc, (unsigned long long)g_gr_prop); + if(g_disk_split) printf("disk-load split: draft %llu + absorb %llu + verify/main %llu misses | " + "MTP-layer %llu loads %.2f GB | main-layers %llu loads %.2f GB (MTP %.1f%% of bytes)\n", + (unsigned long long)m->miss_draft, (unsigned long long)m->miss_absorb, + (unsigned long long)(m->miss - m->miss_draft - m->miss_absorb), + (unsigned long long)m->ld_mtp, m->bytes_mtp/1e9, + (unsigned long long)m->ld_main, m->bytes_main/1e9, + (m->bytes_mtp+m->bytes_main)?100.0*m->bytes_mtp/(m->bytes_mtp+m->bytes_main):0.0); #ifdef COLI_CUDA if(m->gpu_expert_count) printf("CUDA expert tier: %d resident experts (%.2f GB) | %llu calls served from VRAM\n", m->gpu_expert_count,m->gpu_expert_bytes/1e9,(unsigned long long)m->gpu_expert_calls); @@ -2969,12 +4472,21 @@ static void run_text(Model *m, const char *snap, const char *prompt, int ngen){ if(g_pilot_real) printf("PILOT_REAL: %ld load cross-layer completati, %ld scartati (main gia' sul layer) | PILOT_K=%d\n", (long)atomic_load_explicit(&g_pilot_loads,memory_order_relaxed), (long)atomic_load_explicit(&g_pilot_drops,memory_order_relaxed), g_pilot_k); + if(g_pilot_two) printf("PILOT_TWO: two-step shared-expert-corrected prefetch active (3 extra matmuls/prediction)\n"); if(g_looka){ - const char *nm[3]={"previous token (=SPEC prefetch)","layer input, skip attention","next layer (one step ahead)"}; + const char *nm[4]={"previous token (=SPEC prefetch)","layer input, skip attention","next layer (PILOT, stale)","next layer (two-step, shared-expert)"}; printf("LOOKAHEAD routing — recall of true experts in predicted top-8:\n"); - for(int i=0;i<3;i++) printf(" %-38s %5.1f%% (%lld/%lld)\n", nm[i], + for(int i=0;i<4;i++) printf(" %-42s %5.1f%% (%lld/%lld)\n", nm[i], la_tot[i]?100.0*la_hit[i]/la_tot[i]:0.0, (long long)la_hit[i], (long long)la_tot[i]); } + /* TOKENS=1: dump the generated token ids (newline-separated) to stderr, + * for exact A/B comparison across decode paths (e.g. resident vs CPU). + * The ids are all[np .. np+produced-1]. */ + if(getenv("TOKENS") && atoi(getenv("TOKENS"))){ + fprintf(stderr,"[TOKENS] %d generated:",produced); + for(int i=np;ic; int nb=0; for(int l=0;ln_layers;l++){ if(!m->npin || m->npin[l]<1 || !m->eheat[l]) continue; +#ifdef COLI_CUDA + int cold=-1,hot=-1; + for(int z=0;znpin[l];z++){ + ESlot *s=&m->pin[l][z]; uint32_t heat=m->eheat[l][s->eid]; + if(s->g.cuda_eligible){ + if(cold<0||heateheat[l][m->pin[l][cold].eid]) cold=z; + }else if(hot<0||heat>m->eheat[l][m->pin[l][hot].eid]) hot=z; + } + if(cold>=0&&hot>=0){ + uint32_t ch=m->eheat[l][m->pin[l][cold].eid],hh=m->eheat[l][m->pin[l][hot].eid]; + if(hh>ch+1){ + RepinCand v={(long)hh-(long)ch,l,cold,m->pin[l][hot].eid,1}; + if(nbout[w].gain)out[w]=v; } + continue; + } + } +#endif ESlot *P=m->pin[l]; int ids[4096], zp, eu; long g; int np=m->npin[l]; if(np>4096) np=4096; for(int z=0;zeheat[l],m->elast[l],m->eaccess_clock, c->n_experts,ids,np,&zp,&eu,&g)) continue; - if(nbout[w].gain){ out[w].gain=g; out[w].l=l; out[w].slot=zp; out[w].eid=eu; } } + if(g>out[w].gain) out[w]=(RepinCand){g,l,zp,eu,0}; } } return nb; } -static void repin_pass(Model *m){ +static void repin_pass_limit(Model *m,int limit){ if(g_repin<=0) return; if(m->n_emit - g_last_repin < (uint64_t)g_repin) return; g_last_repin = m->n_emit; - RepinCand cd[4]; int nb=repin_pick(m,cd,4); + double pass_t0=now_s(); int gpu_swaps=0; + RepinCand cd[130]; + if(limit<1) limit=1; if(limit>130) limit=130; + int nb=repin_pick(m,cd,limit); +#ifdef COLI_CUDA + /* Cold GPU slots have no host backing. Restore all demoted experts in + * parallel first; serial 20 MB reads made a 32-slot adaptation pass cost + * ~0.7 s on the six-GPU host. */ + #pragma omp parallel for schedule(dynamic,1) + for(int b=0;bpin[cd[b].l][cd[b].slot]; + expert_host_ensure(m,cd[b].l,s); + } + for(int b=0;bpin[cd[b].l][cd[b].slot]; + m->resident_bytes+=qt_bytes(&s->g)+qt_bytes(&s->u)+qt_bytes(&s->d); + } +#endif for(int b=0;bpin[cd[b].l][cd[b].slot]; int old=s->eid; uint32_t old_heat=m->eheat[cd[b].l][old], new_heat=m->eheat[cd[b].l][cd[b].eid]; #ifdef COLI_CUDA + if(cd[b].gpu_swap){ + ESlot *hot=NULL; + for(int z=0;znpin[cd[b].l];z++) + if(m->pin[cd[b].l][z].eid==cd[b].eid){hot=&m->pin[cd[b].l][z];break;} + if(!hot||hot->g.cuda_eligible) continue; + double t0=now_s(); + QT *cq[3]={&s->g,&s->u,&s->d},*hq[3]={&hot->g,&hot->u,&hot->d}; + int ok=1; + for(int k=0;k<3;k++){ + hq[k]->cuda=cq[k]->cuda; cq[k]->cuda=NULL; + hq[k]->cuda_device=cq[k]->cuda_device; + hq[k]->cuda_eligible=1; cq[k]->cuda_eligible=0; + if(!qt_cuda_update(hq[k])) ok=0; + } + if(!ok){ fprintf(stderr,"[REPIN] refresh VRAM fallito\n"); exit(1); } + /* promoted expert now computes from VRAM: drop its host mlock + * (mmap path; no-op otherwise) or every swap leaks locked pages */ + qt_unwire_mmap(&hot->g); qt_unwire_mmap(&hot->u); qt_unwire_mmap(&hot->d); + if(g_cuda_release_host) expert_host_release(m,hot); + gpu_swaps++; + if(getenv("REPIN_VERBOSE")) fprintf(stderr, + "[REPIN] VRAM layer %d: esce/out %d (heat=%u) <- entra/in %d " + "(heat=%u) in %.0f ms\n",cd[b].l,old,old_heat,cd[b].eid,new_heat,(now_s()-t0)*1e3); + continue; + } int gpu=s->g.cuda_eligible; int64_t old_gpu=gpu ? (int64_t)coli_cuda_tensor_bytes(s->g.cuda) +(int64_t)coli_cuda_tensor_bytes(s->u.cuda) @@ -3051,6 +4622,8 @@ static void repin_pass(Model *m){ fprintf(stderr,"[REPIN] %s layer %d: evict %d (heat=%u) <- admit %d (heat=%u) in %.0f ms\n", tier,cd[b].l,old,old_heat,cd[b].eid,new_heat,(now_s()-t0)*1e3); } + if(gpu_swaps) fprintf(stderr,"[REPIN] VRAM: %d expert scambiati/swapped in %.0f ms\n", + gpu_swaps,(now_s()-pass_t0)*1e3); for(int l=0;lc.n_layers;l++) if(m->eheat[l]) tier_decay(m->eheat[l],m->c.n_experts); } /* ---- KV SU DISCO: la conversazione si riapre CALDA (KVSAVE=0 disattiva) ---- @@ -3068,36 +4641,73 @@ static void kv_hdr(Model *m, int32_t *h, int nrec){ h[0]=c->n_layers; h[1]=c->kv_lora; h[2]=c->qk_rope; h[3]=m->has_dsa?c->index_hd:0; h[4]=nic; h[5]=c->vocab; h[6]=nrec; h[7]=0; } +/* Bytes of one on-disk record: [tok i32][Lc+Rc per layer][Ic per DSA layer]. + * Layout matches what kv_disk_append writes and kv_disk_load reads. */ +static int64_t kv_rec_bytes(Model *m){ + Cfg *c=&m->c; + int64_t rec = 4 + (int64_t)c->n_layers*(c->kv_lora+c->qk_rope)*4; + if(m->has_dsa) for(int i=0;in_layers;i++) if(m->Ic[i]) rec+=(int64_t)c->index_hd*4; + return rec; +} +/* Open the persistent handle lazily; write the header if the file is new. After + * this returns successfully, k->disk_fp is valid for the engine's lifetime and + * positioned at end-of-header (nrec==0 case) or wherever the caller seeks. */ +static int kv_disk_open(Model *m){ + KVState *k=m->kv; + if(k->disk_fp) return 1; + k->disk_fp=fopen(k->disk_path,"r+b"); + if(!k->disk_fp){ /* not there yet -> create + header */ + k->disk_fp=fopen(k->disk_path,"wb"); + if(!k->disk_fp) return 0; + int32_t h[8]; kv_hdr(m,h,0); + fwrite(KV_MAGIC,1,8,k->disk_fp); fwrite(h,4,8,k->disk_fp); + fflush(k->disk_fp); + fclose(k->disk_fp); + k->disk_fp=fopen(k->disk_path,"r+b"); /* reopen r+b for append */ + if(!k->disk_fp) return 0; + } + return 1; +} static void kv_disk_truncate(Model *m, int nrec){ if(!g_kvsave) return; KVState *k=m->kv; + if(k->disk_fp){ fclose(k->disk_fp); k->disk_fp=NULL; } /* drop to shrink on disc */ FILE *f=fopen(k->disk_path,"r+b"); if(!f){ k->disk_nrec=0; return; } k->disk_nrec=nrec; - int32_t nr=nrec; fseek(f,8+6*4,SEEK_SET); fwrite(&nr,4,1,f); fclose(f); + int32_t nr=nrec; fseek(f,8+6*4,SEEK_SET); fwrite(&nr,4,1,f); + fflush(f); fclose(f); } static void kv_disk_reset(Model *m){ kv_disk_truncate(m,0); } static void kv_disk_append(Model *m, const int *hist, int len){ KVState *k=m->kv; if(!g_kvsave || len<=k->disk_nrec) return; Cfg *c=&m->c; - FILE *f=fopen(k->disk_path,"r+b"); - if(!f){ f=fopen(k->disk_path,"wb"); if(!f) return; - int32_t h[8]; kv_hdr(m,h,0); fwrite(KV_MAGIC,1,8,f); fwrite(h,4,8,f); } - int64_t rec = 4 + (int64_t)c->n_layers*(c->kv_lora+c->qk_rope)*4; - if(m->has_dsa) for(int i=0;in_layers;i++) if(m->Ic[i]) rec+=(int64_t)c->index_hd*4; + if(!kv_disk_open(m)) return; + FILE *f=k->disk_fp; + int64_t rec = kv_rec_bytes(m); + /* grow the contiguous staging buffer if the record is larger (#1 batching) */ + if(rec > k->disk_buf_cap){ + uint8_t *nb=realloc(k->disk_buf, rec); + if(!nb) return; /* OOM: skip this turn, retry next */ + k->disk_buf=nb; k->disk_buf_cap=rec; + } fseek(f, 8+8*4 + (int64_t)k->disk_nrec*rec, SEEK_SET); for(int p=k->disk_nrec;pdisk_buf; /* pack token + every layer into one record */ + *(int32_t*)b = hist[p]; b+=4; for(int i=0;in_layers;i++){ - fwrite(m->Lc[i]+(int64_t)p*c->kv_lora, 4, c->kv_lora, f); - fwrite(m->Rc[i]+(int64_t)p*c->qk_rope, 4, c->qk_rope, f); + memcpy(b, m->Lc[i]+(int64_t)p*c->kv_lora, (size_t)c->kv_lora*4); b+=c->kv_lora*4; + memcpy(b, m->Rc[i]+(int64_t)p*c->qk_rope,(size_t)c->qk_rope*4); b+=c->qk_rope*4; } - if(m->has_dsa) for(int i=0;in_layers;i++) if(m->Ic[i]) - fwrite(m->Ic[i]+(int64_t)p*c->index_hd, 4, c->index_hd, f); + if(m->has_dsa) for(int i=0;in_layers;i++) if(m->Ic[i]){ + memcpy(b, m->Ic[i]+(int64_t)p*c->index_hd, (size_t)c->index_hd*4); b+=c->index_hd*4; + } + fwrite(k->disk_buf, 1, (size_t)rec, f); /* one fwrite per position (was ~157) */ } fflush(f); /* dati prima, contatore poi */ - int32_t nr=len; fseek(f,8+6*4,SEEK_SET); fwrite(&nr,4,1,f); fclose(f); + int32_t nr=len; fseek(f,8+6*4,SEEK_SET); fwrite(&nr,4,1,f); + fflush(f); /* persist the counter too */ k->disk_nrec=len; } static int kv_disk_load(Model *m, int *hist, int maxctx){ @@ -3152,6 +4762,8 @@ static void serve_ctx_init(Model *m, ServeCtx *s, const char *snap, int slot, in static void serve_ctx_free(Model *m, ServeCtx *s){ KVState *k=&s->kv; int NR=m->c.n_layers+1; + if(k->disk_fp){ fclose(k->disk_fp); k->disk_fp=NULL; } + free(k->disk_buf); k->disk_buf=NULL; if(k->Lc) for(int i=0;iLc[i]); free(k->Rc[i]); } if(k->Ic) for(int i=0;ic.n_layers;i++) free(k->Ic[i]); free(k->Lc); free(k->Rc); free(k->Ic); free(k->kv_start); free(s->hist); @@ -3175,6 +4787,7 @@ static void mux_done(Model *m, ServeCtx *sc, ServeReq *r){ double dt=now_s()-r->started; if(dt<1e-6) dt=1e-6; double dh=(double)(m->hits-r->hits0), dm=(double)(m->miss-r->miss0); hwinfo_emit(m); + usage_save(m); /* la cache che impara non deve aspettare l'uscita */ tiers_emit(m); emap_emit(m); hits_emit(m); @@ -3236,6 +4849,7 @@ static int mux_submit(Model *m, Tok *T, ServeCtx *ctx, ServeReq *req, int nctx, kv_disk_truncate(m,sc->len); } int add=nt-sc->len; if(add>0) memcpy(sc->hist+sc->len,tmp+sc->len,(size_t)add*sizeof(int)); + fprintf(stderr,"[API] KV slot %d prefix %d/%d token, prefill %d\n",sub.slot,sc->len,nt,add); free(tmp); float *logit = add>0 ? step(m,sc->hist+sc->len,add,sc->len) : step(m,sc->hist+sc->len-1,1,sc->len-1); @@ -3264,13 +4878,26 @@ static void run_serve_mux(Model *m, const char *snap){ KVState *initial=m->kv; free(initial->kv_start); free(initial); ServeCtx *ctx=calloc(nctx,sizeof(*ctx)); ServeReq *req=calloc(nctx,sizeof(*req)); for(int i=0;i0) + /* Anonymous pipes are NOT waitable objects: WaitForSingleObject on them is + * undefined (always-signaled or WAIT_FAILED), and PeekNamedPipe fails on + * file/console handles — the old gate never dispatched (#195). New rule: + * idle -> block in getline() inside mux_submit (same semantics as the + * POSIX select(NULL)); active -> poll the pipe with PeekNamedPipe, and on + * non-pipe stdin just defer submits until the batch finishes. */ + if(eof) ready=0; + else if(!active) ready=1; + else ready=(PeekNamedPipe(ih,NULL,0,NULL,&avail,NULL) && avail>0)?1:0; + if(ready) #endif if(mux_submit(m,&T,ctx,req,nctx,maxctx,eos)<0) eof=1; } @@ -3363,9 +4990,11 @@ static void run_serve(Model *m, const char *snap){ #define len (sc->len) #define first (sc->first) char *line=NULL; size_t cap=0; ssize_t nr; char *buf=malloc(1<<16); + intr_install(); /* Ctrl-C = fine turno, non fine processo */ printf("\x01\x01" "READY" "\x01\x01\n"); printf("STAT 0 0.00 0.0 %.2f\n", rss_gb()); fflush(stdout); tiers_emit(m); while((nr=getline(&line,&cap,stdin))>0){ + g_intr=0; /* interruzioni arrivate tra i turni: stantie */ if(nr>0 && line[nr-1]=='\n') line[--nr]=0; if(!strcmp(line,"\x02RESET")){ len=0; first=1; if(m->has_mtp) m->kv_start[m->c.n_layers]=-1; kv_disk_reset(m); @@ -3447,7 +5076,11 @@ static void run_serve(Model *m, const char *snap){ printf("\x01\x01" "END" "\x01\x01\n"); printf("STAT 0 0.00 0.0 %.2f 0 0\n", rss_gb()); fflush(stdout); continue; } first=0; int cur=req_ngen; if(len+k+cur+g_draft+2>=maxctx) cur=maxctx-len-k-g_draft-2; - uint64_t h0=m->hits, ms0=m->miss; double tt0=now_s(); + uint64_t h0=m->hits, ms0=m->miss; + uint64_t rs0=m->route_slots, rw0=m->route_swaps; + uint64_t agh0=m->route_agree_hit, agt0=m->route_agree_tot; + uint64_t kln0=m->route_kl_n; double kls0=m->route_kl_sum; + double tt0=now_s(); float *logit; if(k>0){ logit=step(m,hist+len,k,len); len+=k; } else logit=step(m,hist+len-1,1,len-1); /* prompt identico/prefisso: rigenera i logits */ @@ -3458,9 +5091,21 @@ static void run_serve(Model *m, const char *snap){ else free(logit); double tdt=now_s()-tt0; if(tdt<1e-6) tdt=1e-6; double dh=(double)(m->hits-h0), dm=(double)(m->miss-ms0); + uint64_t rslots=m->route_slots-rs0, rswaps=m->route_swaps-rw0; + double swap_pct=rslots?100.0*rswaps/rslots:0.0; + uint64_t ag_hit=m->route_agree_hit-agh0, ag_tot=m->route_agree_tot-agt0; + uint64_t kl_n=m->route_kl_n-kln0; double kl_sum=m->route_kl_sum-kls0; + double agree_pct=ag_tot?100.0*ag_hit/ag_tot:100.0; + double kl_mean=kl_n?kl_sum/(double)kl_n:0.0; printf("%s\x01\x01" "END" "\x01\x01\n",raw_mode?"":"\n"); - printf("STAT %d %.2f %.1f %.2f %d %d\n", prod, prod/tdt, + printf("STAT %d %.2f %.1f %.2f %d %d", prod, prod/tdt, (dh+dm)>0?100.0*dh/(dh+dm):0.0, rss_gb(), prompt_tokens, prod>=cur); + if(g_cache_route || rslots || ag_tot) + printf(" swap_pct=%.1f route_swaps=%llu route_slots=%llu" + " route_agree=%.1f route_kl=%.4f", + swap_pct,(unsigned long long)rswaps,(unsigned long long)rslots, + agree_pct,kl_mean); + printf("\n"); fflush(stdout); free(raw); g_temp=base_temp; g_nuc=base_nuc; usage_save(m); /* la cache che impara: storia aggiornata a ogni turno */ @@ -3523,27 +5168,47 @@ static void ehit_mark(Model *m, int layer, int eid){ /* HWINFO: hardware snapshot for the web dashboard — emitted once at READY. */ static void hwinfo_emit(Model *m){ - Cfg *c=&m->c; + Cfg *c=&m->c; (void)c; /* silence -Wunused on builds without /proc (#148 report) */ /* CPU */ - char cpu[256]=""; FILE *ci=fopen("/proc/cpuinfo","r"); + char cpu[256]=""; +#ifdef _WIN32 + /* niente /proc su Windows: brand string via CPUID (0x80000002..4), zero + * dipendenze extra. La dashboard mostrava "0 GB RAM / 0 cores" perche' + * tutto questo blocco era solo-Linux mentre il ramo CUDA funzionava. */ +#if defined(__x86_64__) || defined(__i386__) + { unsigned int r[12]={0}; unsigned int *w=r; + for(unsigned int f=0x80000002u; f<=0x80000004u; f++,w+=4) + __get_cpuid(f,&w[0],&w[1],&w[2],&w[3]); + char *b=(char*)r; b[47]=0; while(*b==' ')b++; + snprintf(cpu,sizeof(cpu),"%s",b); } +#endif +#else + FILE *ci=fopen("/proc/cpuinfo","r"); if(ci){ char ln[256]; while(fgets(ln,sizeof(ln),ci)) if(!strncmp(ln,"model name",10)){ char *p=strchr(ln,':'); if(p){ p++; while(*p==' ')p++; int n=(int)strlen(p); if(n>0&&p[n-1]=='\n')p[--n]=0; snprintf(cpu,sizeof(cpu),"%s",p); } break; } fclose(ci); } +#endif int cores=0; -#ifdef _SC_NPROCESSORS_ONLN +#ifdef _WIN32 + { SYSTEM_INFO si; GetSystemInfo(&si); cores=(int)si.dwNumberOfProcessors; } +#elif defined(_SC_NPROCESSORS_ONLN) cores=(int)sysconf(_SC_NPROCESSORS_ONLN); #endif /* RAM */ double ram_total=0,ram_avail=0; +#ifdef _WIN32 + compat_meminfo(&ram_total,&ram_avail); /* GlobalMemoryStatusEx, gia' in compat.h */ +#else FILE *mi=fopen("/proc/meminfo","r"); if(mi){ char ln[256]; double mt=0,ma=0; while(fgets(ln,sizeof(ln),mi)){ if(sscanf(ln,"MemTotal: %lf",&mt)==1) ram_total=mt/1e6; if(sscanf(ln,"MemAvailable: %lf",&ma)==1) ram_avail=ma/1e6; } fclose(mi); } +#endif /* GPU */ int ngpu=0; double vram_total=0; char gpu_name[128]=""; @@ -3688,7 +5353,7 @@ static int mem_should_wire(void){ /* Inchioda [addr,addr+len) in RAM fisica. No-op fuori da POSIX (Windows ecc.). * EN: wire [addr,addr+len) into physical RAM. No-op off POSIX (Windows, etc.). */ static int mem_wire(void *addr, size_t len){ -#if defined(__APPLE__) || defined(__linux__) +#if defined(__APPLE__) || defined(__linux__) || defined(__FreeBSD__) return mlock(addr, len); #elif defined(_WIN32) return compat_mlock(addr, len); /* VirtualLock + working-set growth */ @@ -3698,8 +5363,57 @@ static int mem_wire(void *addr, size_t len){ } /* Inchioda tutti gli slab degli expert pinnati (pesi + scale). Non fatale se fallisce. * EN: wire all pinned-expert slabs (weights + scales). Non-fatal on failure. */ +/* mlock a single mmap'd QT's weight + scale ranges. Skips VRAM-tier QTs + * (cuda_eligible): their compute runs from device memory, so wiring the host + * mmap range would pin ~137 GB of never-touched file pages. NOTE the q8/q4 + * NULL check alone is NOT enough here: expert_host_release() early-returns + * for mmap experts (no slab) without nulling the host pointers, so GPU-tier + * slots keep live-looking q8/q4 forever -- that was the bug that wired 363 GB + * instead of 231 GB and starved the kernel into page-cache thrashing. + * wired/failed are accumulated into the caller's counters. */ +/* undo qt_wire_mmap for one QT: used when a REPIN gpu_swap promotes a wired + * RAM-tier expert into VRAM -- without this every promotion leaks its locked + * host range and the dead-weight lock re-grows over a long session. */ +static void qt_unwire_mmap(QT *t){ + if(!g_mmap || !mem_should_wire()) return; + if(!t->q8 && !t->q4) return; + int64_t scale_b=(int64_t)t->O*4; + int64_t weight_b=qt_bytes(t)-scale_b; + void *wp=t->q8?(void*)t->q8:(void*)t->q4; +#if defined(__APPLE__) || defined(__linux__) || defined(__FreeBSD__) + if(weight_b>0 && !munlock(wp,(size_t)weight_b)) g_mmap_wired-=weight_b; + if(t->s && scale_b>0 && !munlock(t->s,(size_t)scale_b)) g_mmap_wired-=scale_b; +#elif defined(_WIN32) + if(weight_b>0 && !compat_munlock(wp,(size_t)weight_b)) g_mmap_wired-=weight_b; + if(t->s && scale_b>0 && !compat_munlock(t->s,(size_t)scale_b)) g_mmap_wired-=scale_b; +#endif +} +static void qt_wire_mmap(QT *t, int64_t *wired, long *failed){ + if(!t->q8 && !t->q4) return; + if(t->cuda_eligible) return; /* resident in VRAM; host range is dead weight */ + int64_t scale_b=(int64_t)t->O*4; + int64_t weight_b=qt_bytes(t)-scale_b; + void *wp=t->q8?(void*)t->q8:(void*)t->q4; + if(weight_b>0){ if(mem_wire(wp,(size_t)weight_b)==0) *wired+=weight_b; else (*failed)++; } + if(t->s && scale_b>0){ if(mem_wire(t->s,(size_t)scale_b)==0) *wired+=scale_b; else (*failed)++; } +} static void pin_wire(Model *m){ if(!mem_should_wire()) return; + if(g_mmap){ + /* Wire the FINAL resident set only, after pin_load's GPU-upload pass + * has already run -- qt_wire_mmap() skips cuda_eligible (VRAM-tier) + * slots, so only the genuinely RAM-tier experts get locked. */ + Cfg *c=&m->c; double t0=now_s(); + for(int i=0;in_layers;i++) for(int z=0;znpin[i];z++){ + ESlot *s=&m->pin[i][z]; + qt_wire_mmap(&s->g,&g_mmap_wired,&g_mmap_wire_failed); + qt_wire_mmap(&s->u,&g_mmap_wired,&g_mmap_wire_failed); + qt_wire_mmap(&s->d,&g_mmap_wired,&g_mmap_wire_failed); + } + fprintf(stderr,"[PIN] mlock (mmap): %.1f GB wired in physical RAM%s in %.0fs\n", + g_mmap_wired/1e9, g_mmap_wire_failed?" (some allocations failed -- raise: ulimit -l unlimited)":"", now_s()-t0); + return; + } Cfg *c=&m->c; double t0=now_s(); int64_t wired=0; long failed=0; for(int i=0;in_layers;i++) for(int z=0;znpin[i];z++){ ESlot *s=&m->pin[i][z]; @@ -3719,6 +5433,7 @@ typedef struct { int l,e; uint32_t c; } PinRec; static int pin_rec_cmp(const void *a,const void *b){ const PinRec *x=a,*y=b; return x->cc?1:x->c>y->c?-1:0; } +static double expert_avail(Model *m, double ram_gb, int ebits, int max_ctx); /* def. sotto */ static void pin_load(Model *m, const char *statspath, double gb){ FILE *f=fopen(statspath,"r"); if(!f){ perror(statspath); return; } Cfg *c=&m->c; int cap=(c->n_layers+1)*c->n_experts; @@ -3744,7 +5459,19 @@ static void pin_load(Model *m, const char *statspath, double gb){ free(seen); qsort(r,(size_t)n,sizeof(*r),pin_rec_cmp); int64_t eb=expert_bytes_probe(m,m->ebits); - int npin=(int)(gb*1e9/eb); if(npin>n) npin=n; + /* PIN_GB=all (#80): NON "tutti" alla lettera. Pinnare l'intero set ignora il + * budget --ram e fa OOM-kill del kernel a meta' generazione (#229: host 92 GB + * ucciso con --ram 78, anon-rss 89 GB). Clampa a quanti expert entrano nel + * budget RAM, come AUTOPIN; il pin aggiorna resident_bytes, quindi cap_for_ram + * dopo restringe la LRU di conseguenza (nessun doppio conteggio). */ + int npin; + if(gb<0){ + double ram_env=getenv("RAM_GB")?atof(getenv("RAM_GB")):0.0; + int est_ctx=getenv("CTX")?atoi(getenv("CTX")):4096; /* stesso default del call site */ + double avail=expert_avail(m,ram_env,m->ebits,est_ctx); + npin=avail>0?(int)(avail/eb):0; + } else npin=(int)(gb*1e9/eb); + if(npin>n) npin=n; if(npin<1){ free(r); return; } int *cnt_l=calloc(c->n_layers+1,sizeof(int)); /* +1: riga MTP */ for(int a=0;a0) for(int i=0;i0||g_cuda_expert_auto)) for(int i=0;isafe_total) budget=safe_total; + if(g_cuda_expert_auto||budget>safe_total) budget=safe_total; if(g_cuda_enabled&&g_cuda_release_host&&budget>0){ gpu_prefix=(int)(budget/eb)+g_cuda_ndev; if(gpu_prefix>npin)gpu_prefix=npin; } #else int gpu_prefix=0; @@ -3776,7 +5503,7 @@ static void pin_load(Model *m, const char *statspath, double gb){ expert_load(m,r[a].l,r[a].e,&m->pin[r[a].l][slot_of[a]],1); m->resident_bytes+=(int64_t)(gpu_prefix?gpu_prefix:npin)*eb; #ifdef COLI_CUDA - if(g_cuda_enabled && g_cuda_expert_gb>0){ + if(g_cuda_enabled && budget>0){ int gpu_limit=gpu_prefix?gpu_prefix:npin; for(int a=0;agpu_expert_bytesc; int64_t eb=expert_bytes_probe(m,ebits); if(ram_gb<=0){ ram_gb=g_mem_avail_boot*0.88; if(ram_gb<4) ram_gb=8; } - double slack = 1.2e9 + 2.5e9 + 64.0*(double)eb + double ws_b = (g_expert_budget>0 && g_expert_budget<64) ? (double)(g_expert_budget+4)*(double)eb : 64.0*(double)eb; + double slack = 1.2e9 + 2.5e9 + ws_b + kv_pool_bytes(m,max_ctx) + (double)max_ctx*c->n_heads*(c->qk_nope+c->v_head)*4.0; return ram_gb*1e9 - (double)m->resident_bytes - slack; @@ -3889,16 +5617,50 @@ static void cap_for_ram(Model *m, double ram_gb, int ebits, int max_ctx){ * KV cache a max_ctx, kvb_all della ricostruzione k/v in attention, * attivazioni+logits+overhead ~1.2 GB */ double ws_b = 64.0*(double)eb; + /* Under EXPERT_BUDGET, the block-of-64 working set is capped at budget experts + * per layer — only ws[0..budget-1] are populated, not all 64. The 64×eb reserve + * overcounts by 16x at budget=4, starving the LRU cache (cap 3 instead of 4). + * Cap=4 matches budget=4, eliminating LRU thrashing that causes excessive disk + * re-reads. Clamp ws_b to the actual budget (min 8 for non-budgeted / prefill). */ + if(g_expert_budget>0 && g_expert_budget<64) ws_b = (double)(g_expert_budget+4) * (double)eb; double kv_b = kv_pool_bytes(m,max_ctx); double kvb_b = (double)max_ctx*c->n_heads*(c->qk_nope+c->v_head)*4.0; - /* RISERVA PAGE-CACHE (misurato 2026-07-06): strangolarla fa crollare le pread - * buffered da ~800 a ~180 MB/s — gli ultimi GB di LRU rendono MENO di quanto - * costino in banda disco persa. 2.5 GB restano SEMPRE al kernel. */ + /* RISERVA PAGE-CACHE (misurato 2026-07-06 su Linux): strangolarla fa crollare + * le pread buffered da ~800 a ~180 MB/s — gli ultimi GB di LRU rendono MENO di + * quanto costino in banda disco persa. 2.5 GB restano SEMPRE al kernel. + * NOTE: tested removing this under Windows+DIRECT (it should be dead weight when + * O_DIRECT bypasses the buffer cache). Result: cap went 4->5 but RSS hit 24 GB + * on a 32 GB machine, causing memory pressure that DROPPED the hit rate (73%->57%) + * and slowed decode (1.03->0.83 tok/s). The reserve is a legitimate safety margin + * for OS + CUDA + file metadata, not just buffered pread throughput. Keep it. */ double pc_b = 2.5e9; double slack = 1.2e9 + pc_b + ws_b + kv_b + kvb_b; double avail = ram_gb*1e9 - (double)m->resident_bytes - slack; int capmax = (avail>0 && nsp>0) ? (int)(avail/((double)nsp*eb)) : 0; + int floored = capmax<1; /* il budget non regge nemmeno UNO slot per layer */ if(capmax<1) capmax=1; + /* Il floor a 1 e' una bugia comoda: con avail negativo capmax sarebbe 0, cioe' + * "non ci sto nel tuo budget". Alzarlo a 1 e proseguire trasforma "non ci sto" + * in "sforo" -- ed e' esattamente l'OOM-kill a meta' generazione che questa + * funzione esiste per evitare. Il kernel uccide con SIGKILL: nessun errore, + * nessun log, il motore muore muto (issue #305). Dirlo, e fermarsi se il picco + * non entra nemmeno nella RAM realmente disponibile misurata all'avvio. */ + if(floored){ + double peak = (double)m->resident_bytes + (double)capmax*nsp*eb + slack; + fprintf(stderr,"[RAM_GB=%.1f%s] WARNING: cap=1 is the floor, projected peak %.1f GB is " + "%.1f GB OVER the budget (resident %.1f GB + reserve %.1f GB).%s\n", + ram_gb,auto_b?" auto":"",peak/1e9,(peak-ram_gb*1e9)/1e9, + m->resident_bytes/1e9,slack/1e9, + getenv("PIN_GB")?" PIN_GB is inflating the resident set: lower it or drop it.":""); + if(g_mem_avail_boot>0 && peak > g_mem_avail_boot*1e9 && + !(getenv("COLI_RAM_OVERCOMMIT") && atoi(getenv("COLI_RAM_OVERCOMMIT")))){ + fprintf(stderr,"[RAM] refusing to start: that peak also exceeds the %.1f GB actually " + "available on this machine, so the kernel would OOM-kill this run mid-generation.\n" + "[RAM] lower PIN_GB, lower the context, or raise the RAM budget if the box really has it " + "(COLI_RAM_OVERCOMMIT=1 overrides this check).\n", g_mem_avail_boot); + exit(2); + } + } if(capmax < m->ecap){ fprintf(stderr,"[RAM_GB=%.1f%s] resident %.1f GB + reserve %.1f GB (ws %.1f, KV %dx%d %.1f, kvb %.1f), " "experts %.1f MB x %d layers -> cap lowered %d->%d (projected peak %.1f GB)\n", @@ -3931,6 +5693,24 @@ static void cap_for_ram(Model *m, double ram_gb, int ebits, int max_ctx){ } } +/* The user's generation prompt. COLI_PROMPT is honored on every platform; a bare + * PROMPT is honored too, EXCEPT on Windows, where cmd.exe always exports its own + * PROMPT template (default "$P$G", the thing that draws "C:\...>") into the child's + * environment. That is a shell UI string, not a prompt: taking it would send the + * engine into text-generation mode (needing a tokenizer) instead of the oracle + * self-test, and would "generate" from "$P$G". So on Windows a PROMPT carrying + * cmd's $-metacodes is ignored; set COLI_PROMPT to pass a real prompt from cmd. */ +static const char *coli_user_prompt(void){ + const char *p = getenv("COLI_PROMPT"); + if(p) return p; + p = getenv("PROMPT"); +#ifdef _WIN32 + if(p) for(const char *q=p; q[0]; q++) + if(q[0]=='$' && q[1] && strchr("ABCDEFGHLNPQSTV_+|$", q[1]&~0x20)){ p=NULL; break; } +#endif + return p; +} + int main(int argc, char **argv){ /* ---- Permanent OpenMP hot-thread tuning. The per-expert matmul regions are * tiny and back-to-back; with the default passive wait policy libgomp parks @@ -3962,6 +5742,11 @@ int main(int argc, char **argv){ fprintf(stderr,"[OMP] hot-thread tuning: re-exec once (COLI_NO_OMP_TUNE=1 to skip)\n"); execv("/proc/self/exe", argv); /* returns only on failure -> fall through and run untuned */ perror("[OMP] execv self-reexec failed, running untuned"); +#endif +#ifdef __FreeBSD__ + fprintf(stderr,"[OMP] hot-thread tuning: re-exec once (COLI_NO_OMP_TUNE=1 to skip)\n"); + execv("/proc/curproc/file", argv); /* returns only on failure -> fall through and run untuned */ + perror("[OMP] execv self-reexec failed, running untuned"); #endif } #if defined(__AVX512F__) && defined(__AVX512BW__) @@ -3979,6 +5764,49 @@ int main(int argc, char **argv){ if(g_mmap) fprintf(stderr,"[MMAP] expert = viste zero-copy nei file (page cache = cache)\n"); g_topk = getenv("TOPK")?atoi(getenv("TOPK")):0; g_topp = getenv("TOPP")?atof(getenv("TOPP")):0; + /* EXPERT_BUDGET e' sotto quarantena: la finestra operativa e' misurata VUOTA. + * @bokiko su tre host (#303) e riprodotto qui su un 25 GB / WSL: + * - hellaswag 30% a budget=8 contro 90% a budget spento (25% = il caso); + * - a budget=4 il decode e' rumore ("The **1...: s2151:"); + * - accettazione MTP 0%: quali expert sopravvivono al cap dipende dalla + * residenza in cache al momento del forward, quindi draft e verify NON + * calcolano la stessa funzione -- la stessa invariante che #294 ha appena + * stabilito, violata via stato di cache invece che via scelta del kernel; + * - 0.13 tok/s contro 0.30 di baseline, con 14.66 expert caricati per layer + * contro topk=8: il cap fa piu' I/O di quello che dice di risparmiare, e la + * riga "~N GB I/O saved" conta esperti scartati, non byte non letti. + * Resta compilato e sviluppabile (EXPERT_BUDGET_EXPERIMENTAL=1) perche' l'idea + * -- MoE-Spec, arXiv 2602.16052 -- non e' sbagliata: e' l'implementazione che + * finora non ha un punto in cui sia insieme piu' veloce e corretta. Riaccenderlo + * di default richiede una misura di qualita' accanto a quella di velocita'. */ + g_expert_budget = getenv("EXPERT_BUDGET")?atoi(getenv("EXPERT_BUDGET")):0; + if(g_expert_budget>0 && !getenv("EXPERT_BUDGET_EXPERIMENTAL")){ + fprintf(stderr,"[EXPERT_BUDGET] ignored: measured empty operating window (issue #303).\n" + "[EXPERT_BUDGET] every tested setting is either no faster or no longer coherent:\n" + "[EXPERT_BUDGET] budget=8 -> hellaswag 30%% (90%% with it off) | budget=4 -> decode is noise\n" + "[EXPERT_BUDGET] MTP acceptance 0%% (the cap breaks the draft/verify contract, #294)\n" + "[EXPERT_BUDGET] 0.13 tok/s vs 0.30 baseline, loading 14.7 experts/layer vs topk=8\n" + "[EXPERT_BUDGET] set EXPERT_BUDGET_EXPERIMENTAL=1 to run it anyway (expect garbage).\n"); + g_expert_budget=0; + } + g_cache_route = getenv("CACHE_ROUTE")?atoi(getenv("CACHE_ROUTE")):0; + g_route_j = getenv("ROUTE_J")?atoi(getenv("ROUTE_J")):2; + g_route_m = getenv("ROUTE_M")?atoi(getenv("ROUTE_M")):12; + g_route_p = getenv("ROUTE_P")?atof(getenv("ROUTE_P")):0; + g_route_alpha = getenv("ROUTE_ALPHA")?atof(getenv("ROUTE_ALPHA")):1.f; + g_route_agree = getenv("ROUTE_AGREE")?atoi(getenv("ROUTE_AGREE")):0; + if(g_route_j<0) g_route_j=0; + if(g_route_m<1) g_route_m=1; + if(g_route_m>4096) g_route_m=4096; + if(g_route_alpha<=0.f) g_route_alpha=1.f; + if(g_route_alpha>1.f) g_route_alpha=1.f; + if(g_cache_route) + fprintf(stderr,"[CACHE_ROUTE] on J=%d M=%d P=%.2f alpha=%.2f (pin∪LRU prefer; never default)\n", + g_route_j,g_route_m,g_route_p,g_route_alpha); + if(g_route_agree) + fprintf(stderr,"[ROUTE_AGREE] telemetry on (overlap%% + mean KL vs true top-K)\n"); + /* Auto-enable agree telemetry when CACHE_ROUTE is on (cheap quality leading indicator). */ + if(g_cache_route && !getenv("ROUTE_AGREE")) g_route_agree=1; const char *policy=getenv("COLI_POLICY"); if(!policy) policy="quality"; int experimental=!strcmp(policy,"experimental-fast"); if(strcmp(policy,"quality")&&strcmp(policy,"balanced")&&!experimental){ @@ -3995,17 +5823,46 @@ int main(int argc, char **argv){ g_pilot = getenv("PILOT")?atoi(getenv("PILOT")):0; /* 1 = prefetch pilotato dal router */ g_pilot_real = getenv("PILOT_REAL")?atoi(getenv("PILOT_REAL")):0; /* default OFF: load VERI cross-layer (value-preserving prefetch); PILOT_REAL=1 opta in */ if(g_pilot_real) g_pilot=1; /* PILOT_REAL implica il pilota attivo */ + g_pilot_two = getenv("PILOT_TWO")?atoi(getenv("PILOT_TWO")):0; /* 1 = two-step: shared-expert-corrected router prediction (+2.3% recall, 3 extra matmuls) */ + if(g_pilot_two) g_pilot=1; /* PILOT_TWO implies PILOT active */ /* Default K: hint-only PILOT keeps 8 (WILLNEED hints are free, no eviction). * Under PILOT_REAL the speculative loads are REAL and create LRU eviction * pressure, so at ~28% mispredict a large K thrashes the cache — default to 6 * (best-measured this session) unless the user set PILOT_K explicitly. */ g_pilot_k = getenv("PILOT_K")?atoi(getenv("PILOT_K")):(g_pilot_real?6:8); if(g_pilot_k<1) g_pilot_k=1; - g_pipe = getenv("PIPE")?atoi(getenv("PIPE")):0; /* default OFF: overlap expert load ‖ matmul (byte-identical; reorders I/O). PIPE=1 opts in */ + g_disk_split = getenv("DISK_SPLIT")?atoi(getenv("DISK_SPLIT")):0; /* 1 = split dei disk load nelle stats */ + g_pipe = getenv("PIPE")?atoi(getenv("PIPE")): +#ifdef _WIN32 + 1 /* default ON: overlap expert load ‖ matmul (byte-identical; reorders I/O). PIPE=0 opts out */ +#else + 0 +#endif + ; g_pipe_nw = getenv("PIPE_WORKERS")?atoi(getenv("PIPE_WORKERS")):8; /* I/O worker threads */ if(g_pipe_nw<1) g_pipe_nw=1; g_direct = getenv("DIRECT")?atoi(getenv("DIRECT")):0; + g_uring = getenv("URING")?atoi(getenv("URING")):0; + if(g_uring){ +#ifdef __linux__ + if(g_mmap){ fprintf(stderr,"URING=1 is incompatible with COLI_MMAP=1\n"); return 2; } + g_pipe=1; + if(uring_batch_init(&g_ub_pipe) || (g_pilot_real&&uring_batch_init(&g_ub_pilot))){ + fprintf(stderr,"URING=1: io_uring_setup failed: %s\n",strerror(errno)); return 2; + } + unsigned uw=(unsigned)(g_pipe_nw>64?64:g_pipe_nw); + if(coli_uring_set_workers(&g_ub_pipe.ring,uw) || + (g_pilot_real&&coli_uring_set_workers(&g_ub_pilot.ring,uw))) + fprintf(stderr,"[URING] warning: cannot set io-wq workers=%u: %s\n",uw,strerror(errno)); + fprintf(stderr,"[URING] queued expert I/O active (depth=%d, workers=%u, %s%s)\n",URING_REQ_MAX,uw, + g_direct?"DIRECT=1":"buffered",g_pilot_real?", batched PILOT_REAL":""); + if(!g_direct) fprintf(stderr,"[URING] cold NVMe: DIRECT=1 avoids page-cache copy/readahead bottlenecks\n"); +#else + fprintf(stderr,"URING=1 is supported only on Linux\n"); return 2; +#endif + } g_idot = getenv("IDOT")?atoi(getenv("IDOT")):1; /* 0 = kernel f32 esatti (A/B) */ + g_spec_pin = getenv("SPEC_PIN")?atoi(getenv("SPEC_PIN")):1; /* #163: 0 = gate S-dipendenti storici / legacy S-dependent gates */ if(getenv("ROUTE_TRACE")&&*getenv("ROUTE_TRACE")){ g_route_fp=fopen(getenv("ROUTE_TRACE"),"w"); if(!g_route_fp) fprintf(stderr,"[ROUTE_TRACE] cannot open %s\n",getenv("ROUTE_TRACE")); @@ -4014,6 +5871,11 @@ int main(int argc, char **argv){ g_repin = getenv("REPIN")?atoi(getenv("REPIN")):0; /* RFC: re-pin ogni n token emessi (0=off) / live re-pin every n emitted tokens (0=off) */ g_absorb = getenv("ABSORB")?atoi(getenv("ABSORB")):-1; /* -1 auto: assorbita per S<=4 */ g_dsa_force = getenv("DSA_FORCE")?atoi(getenv("DSA_FORCE")):0; + /* matmul_qt documenta la soglia int4-IDOT come "configurabile con I4S" ma il getenv non + * c'era: la variabile non aveva alcun effetto. I4S= -> IDOT int4 solo per S>=n. + * EN: matmul_qt documents the int4 IDOT threshold as "configurable via I4S", but the + * getenv was missing, so the knob did nothing. I4S= -> int4 IDOT only for S>=n. */ + if(getenv("I4S")) g_i4s=atoi(getenv("I4S")); g_temp = getenv("TEMP")?atof(getenv("TEMP")):-1; /* -1 = auto (1.0 chat/testo, greedy altrove) */ g_nuc = getenv("NUCLEUS")?atof(getenv("NUCLEUS")):0.90f; /* piu' stretto dell'ufficiale 0.95: la coda int4 e' rumore */ if(getenv("SEED")) g_rng = (uint64_t)atoll(getenv("SEED"))*0x9E3779B97F4A7C15ULL+1; @@ -4037,11 +5899,16 @@ int main(int argc, char **argv){ if(!g_cuda_enabled){ fprintf(stderr,"[CUDA] requested backend is unavailable\n"); return 2; } } g_cuda_dense=getenv("CUDA_DENSE")?atoi(getenv("CUDA_DENSE")):0; - g_cuda_expert_gb=getenv("CUDA_EXPERT_GB")?atof(getenv("CUDA_EXPERT_GB")):0; + g_cuda_pipe=getenv("COLI_CUDA_PIPE")?atoi(getenv("COLI_CUDA_PIPE")):0; + const char *cuda_expert=getenv("CUDA_EXPERT_GB"); + g_cuda_expert_auto=cuda_expert&&!strcmp(cuda_expert,"auto"); + g_cuda_expert_gb=cuda_expert&&!g_cuda_expert_auto?atof(cuda_expert):0; + if(!getenv("REPIN")&&g_cuda_expert_auto&&getenv("PIN_GB")&& + !strcmp(getenv("PIN_GB"),"all")) g_repin=16; g_cuda_release_host=getenv("CUDA_RELEASE_HOST")?atoi(getenv("CUDA_RELEASE_HOST")):(g_cuda_ndev>1); if((getenv("COLI_GPU")||getenv("COLI_GPUS"))&&!g_cuda_enabled){ fprintf(stderr,"COLI_GPU(S) requires COLI_CUDA=1\n"); return 2; } if(g_cuda_dense&&!g_cuda_enabled){ fprintf(stderr,"CUDA_DENSE requires COLI_CUDA=1\n"); return 2; } - if(g_cuda_expert_gb>0 && !g_cuda_enabled){ fprintf(stderr,"CUDA_EXPERT_GB requires COLI_CUDA=1\n"); return 2; } + if((g_cuda_expert_gb>0||g_cuda_expert_auto) && !g_cuda_enabled){ fprintf(stderr,"CUDA_EXPERT_GB requires COLI_CUDA=1\n"); return 2; } if(g_cuda_enabled) fprintf(stderr,"[CUDA] mode: routed experts%s%s\n", g_cuda_dense?" + resident dense tensors":" only (resident dense on CPU)", g_cuda_release_host?"; VRAM experts without host backing":""); @@ -4049,7 +5916,8 @@ int main(int argc, char **argv){ if((getenv("COLI_CUDA") && atoi(getenv("COLI_CUDA"))) || getenv("COLI_GPU") || getenv("COLI_GPUS") || (getenv("CUDA_DENSE") && atoi(getenv("CUDA_DENSE"))) || - (getenv("CUDA_EXPERT_GB") && atof(getenv("CUDA_EXPERT_GB"))>0)){ + (getenv("CUDA_EXPERT_GB") && + (!strcmp(getenv("CUDA_EXPERT_GB"),"auto")||atof(getenv("CUDA_EXPERT_GB"))>0))){ fprintf(stderr,"CUDA was requested, but this binary is CPU-only; rebuild with: make CUDA=1\n"); return 2; } @@ -4073,7 +5941,16 @@ int main(int argc, char **argv){ Model m; double t0=now_s(); model_init(&m,snap,cap,ebits,dbits); if(g_draft<0){ #ifdef COLI_CUDA - g_draft = (m.has_mtp&&!g_cuda_enabled) ? 3 : 0; + /* MTP is disabled under CUDA by default: cold (streaming) experts still + * run on the CPU, where the S==1 fused-pair kernel and the S>=2 IDOT + * kernel diverge in FP accumulation order, collapsing draft acceptance + * (#163). GPU-resident experts have no divergence, but the cold subset + * always exists on a single 16 GB card. COLI_CUDA_MTP=1 opts in for + * users who want to test speculation under CUDA — the #163 thread shows + * acceptance can still reach 30-50% even with the cold-expert mismatch. + * See #292 for the diagnostic sweep that identified this. */ + int cuda_mtp = getenv("COLI_CUDA_MTP") ? atoi(getenv("COLI_CUDA_MTP")) : 0; + g_draft = (m.has_mtp && (!g_cuda_enabled || cuda_mtp)) ? 3 : 0; #else g_draft = m.has_mtp ? 3 : 0; #endif @@ -4089,8 +5966,30 @@ int main(int argc, char **argv){ " Keep it on ext4 (for example, /home/...) for memory efficiency and speed.\n", snap); /* HOT-STORE: PIN= [PIN_GB=g] -> top expert per frequenza fissi in RAM. * Va PRIMA di cap_for_ram: i pinnati contano nel residente. */ - if(getenv("PIN")) pin_load(&m, getenv("PIN"), getenv("PIN_GB")?atof(getenv("PIN_GB")):10.0); - if(getenv("COUPLE")&&*getenv("COUPLE")){ /* coupling-scored cross-layer prefetch */ + if(getenv("PIN")){ + const char *pin=getenv("PIN"); char pauto[2100]; + if(!strcmp(pin,"auto")){ + /* PIN=auto: la storia VIVA /.coli_usage (appesa a ogni turno) batte il + * profilo congelato stats.txt — il pin di ogni riavvio riflette il carico reale + * accumulato, non il prompt di bootstrap. Fallback stats.txt per una dir vergine; + * nessuno dei due -> nessun pin (AUTOPIN piu' sotto resta escluso: PIN e' settato). + * EN: prefer the live usage history over the frozen one-shot profile, so each + * reload's pin placement follows the accumulated real workload. */ + snprintf(pauto,sizeof(pauto),"%s/.coli_usage",snap); + FILE *pf=fopen(pauto,"rb"); long psz=0; + if(pf){ fseek(pf,0,SEEK_END); psz=ftell(pf); fclose(pf); } + if(psz<=0){ snprintf(pauto,sizeof(pauto),"%s/stats.txt",snap); + pf=fopen(pauto,"rb"); psz=0; + if(pf){ fseek(pf,0,SEEK_END); psz=ftell(pf); fclose(pf); } } + if(psz>0){ pin=pauto; fprintf(stderr,"[PIN] auto: seeding from %s\n",pauto); } + else { pin=NULL; fprintf(stderr,"[PIN] auto: no .coli_usage or stats.txt in %s yet (no pin this run)\n",snap); } + } + if(pin){ + const char *pin_gb=getenv("PIN_GB"); + pin_load(&m,pin,pin_gb&&!strcmp(pin_gb,"all")?-1.0:pin_gb?atof(pin_gb):10.0); /* PIN_GB=all (#80) */ + } + } + if(getenv("COUPLE")&&*getenv("COUPLE")){ /* coupling-scored cross-layer prefetch (#176) */ g_couple_k=getenv("COUPLE_K")?atoi(getenv("COUPLE_K")):8; if(g_couple_k<1)g_couple_k=1; if(g_couple_k>32)g_couple_k=32; g_couple_d=getenv("COUPLE_D")?atoi(getenv("COUPLE_D")):1; @@ -4120,7 +6019,7 @@ int main(int argc, char **argv){ const char *stats=getenv("STATS"); /* STATS= -> istogramma uso expert a fine run */ /* modo scoring per benchmark: SCORE= -> log-likelihood per riga */ - if(getenv("SCORE")){ run_score(&m, getenv("SCORE")); if(stats) stats_dump(&m,stats); return 0; } + if(getenv("SCORE")){ run_score(&m, snap, getenv("SCORE")); if(stats) stats_dump(&m,stats); return 0; } /* modo serve persistente per la CLI 'coli': SERVE=1 */ if(getenv("SERVE")){ @@ -4130,9 +6029,10 @@ int main(int argc, char **argv){ } /* modo testo reale: PROMPT="..." [NGEN=n] -> tokenizza, genera, detokenizza */ - if(getenv("PROMPT")){ + const char *user_prompt = coli_user_prompt(); /* ignores cmd.exe's PROMPT template (#271) */ + if(user_prompt){ int ngen=getenv("NGEN")?atoi(getenv("NGEN")):64; - run_text(&m, snap, getenv("PROMPT"), ngen); + run_text(&m, snap, user_prompt, ngen); if(stats) stats_dump(&m,stats); return 0; } @@ -4152,10 +6052,17 @@ int main(int argc, char **argv){ * Non e' un bug del motore — vedi #76. */ { int maxid=0; for(int i=0;imaxid) maxid=full[i]; if(m.c.vocab>1000 && maxid<1000 && !getenv("REF_FORCE")){ - fprintf(stderr,"ERRORE: ref_glm.json e' l'oracolo del modello TINY (token max %d, ma il tuo vocab e' %d).\n" - " Self-test motore: SNAP=./glm_tiny TF=1 ./glm 64 16 16 (atteso 32/32)\n" - " Prova reale: PROMPT=\"Ciao\" NGEN=32 SNAP= ./glm 64\n" - " REF_FORCE=1 per eseguire comunque il confronto (senza senso).\n", maxid, m.c.vocab); + fprintf(stderr, + "ERROR: no PROMPT given, so this is oracle self-test mode — but ref_glm.json is the TINY\n" + " model's oracle (max token %d) and your model's vocab is %d. Nothing to validate here.\n" + " Engine self-test: SNAP=./glm_tiny TF=1 ./glm 64 16 16 (expect 32/32)\n" + " Real generation: PROMPT=\"Hello\" NGEN=32 SNAP= ./glm 64\n" + " or: python coli chat --model \n" + " REF_FORCE=1 to run the comparison anyway (meaningless).\n" + " --- IT ---\n" + " Nessun PROMPT: modo auto-validazione, ma ref_glm.json e' l'oracolo del modello TINY\n" + " (token max %d, il tuo vocab e' %d). Usa PROMPT=... per generare davvero (vedi sopra).\n", + maxid, m.c.vocab, maxid, m.c.vocab); return 1; } } @@ -4202,9 +6109,9 @@ int main(int argc, char **argv){ if(g_cuda_enabled) cuda_stats_print(); #endif if(g_looka){ - const char *nm[3]={"previous token (=SPEC prefetch)","layer input, skip attention","next layer (one step ahead)"}; + const char *nm[4]={"previous token (=SPEC prefetch)","layer input, skip attention","next layer (PILOT, stale)","next layer (two-step, shared-expert)"}; printf("LOOKAHEAD routing — recall of true experts in predicted top-8:\n"); - for(int i=0;i<3;i++) printf(" %-38s %5.1f%% (%lld/%lld)\n", nm[i], + for(int i=0;i<4;i++) printf(" %-42s %5.1f%% (%lld/%lld)\n", nm[i], la_tot[i]?100.0*la_hit[i]/la_tot[i]:0.0, (long long)la_hit[i], (long long)la_tot[i]); } if(stats) stats_dump(&m,stats); diff --git a/c/json.h b/c/json.h index 3e1c13d..8a35e24 100644 --- a/c/json.h +++ b/c/json.h @@ -26,8 +26,14 @@ typedef struct { const char *s; char *arena; /* buffer per le stringhe smontate */ size_t acap, aoff; + int depth; /* annidamento corrente: bound contro lo stack-overflow + * da JSON malevolo tipo [[[[...]]]] (discesa ricorsiva) */ } jparser; +/* tetto di annidamento: gli header safetensors / config sono piatti (profondita' + * ~3). 1024 e' larghissimo per input legittimi e ben sotto il limite di stack. */ +#define J_MAX_DEPTH 1024 + static char *j_dup(jparser *p, const char *b, int n) { /* ogni stringa ha la sua allocazione: un'arena con realloc sposterebbe il * buffer invalidando i puntatori gia' emessi (use-after-free). */ @@ -91,10 +97,11 @@ static jval *j_parse_val(jparser *p) { char c = *p->s; if (c == '"') { jval *v = j_new(J_STR); v->str = j_parse_str_raw(p); return v; } if (c == '{') { + if (++p->depth > J_MAX_DEPTH) { p->depth--; return j_new(J_NULL); } p->s++; jval *v = j_new(J_OBJ); int cap = 8; v->keys = malloc(cap * sizeof(char*)); v->kids = malloc(cap * sizeof(jval*)); j_ws(p); - if (*p->s == '}') { p->s++; return v; } + if (*p->s == '}') { p->s++; p->depth--; return v; } for (;;) { j_ws(p); char *key = j_parse_str_raw(p); @@ -107,13 +114,15 @@ static jval *j_parse_val(jparser *p) { if (*p->s == '}') { p->s++; break; } break; } + p->depth--; return v; } if (c == '[') { + if (++p->depth > J_MAX_DEPTH) { p->depth--; return j_new(J_NULL); } p->s++; jval *v = j_new(J_ARR); int cap = 8; v->kids = malloc(cap * sizeof(jval*)); j_ws(p); - if (*p->s == ']') { p->s++; return v; } + if (*p->s == ']') { p->s++; p->depth--; return v; } for (;;) { jval *val = j_parse_val(p); if (v->len == cap) { cap *= 2; v->kids = realloc(v->kids, cap*sizeof(jval*)); } @@ -123,6 +132,7 @@ static jval *j_parse_val(jparser *p) { if (*p->s == ']') { p->s++; break; } break; } + p->depth--; return v; } if (c == 't') { p->s += 4; jval *v = j_new(J_BOOL); v->boolean = 1; return v; } @@ -134,7 +144,7 @@ static jval *j_parse_val(jparser *p) { /* API */ static jval *json_parse(const char *text, char **arena_out) { - jparser p = { text, NULL, 0, 0 }; + jparser p = { text, NULL, 0, 0, 0 }; jval *v = j_parse_val(&p); if (arena_out) *arena_out = p.arena; else free(p.arena); return v; diff --git a/c/olmoe.c b/c/olmoe.c index 1c5ffaa..5923bde 100644 --- a/c/olmoe.c +++ b/c/olmoe.c @@ -12,7 +12,7 @@ #include #include #include -#if defined(__APPLE__) || defined(__linux__) +#if defined(__APPLE__) || defined(__linux__) || defined(__FreeBSD__) #include #endif #include "st.h" diff --git a/c/openai_server.py b/c/openai_server.py index c88888d..ba19e43 100644 --- a/c/openai_server.py +++ b/c/openai_server.py @@ -410,8 +410,11 @@ def generation_options(body, limit): top_p = body.get("top_p") temperature = 0.7 if temperature is None else temperature top_p = 0.9 if top_p is None else top_p - if isinstance(maximum, bool) or not isinstance(maximum, int) or not 1 <= maximum <= limit: - raise APIError(400, f"`{maximum_param}` must be an integer between 1 and {limit}.", maximum_param) + if isinstance(maximum, bool) or not isinstance(maximum, int) or maximum < 1: + raise APIError(400, f"`{maximum_param}` must be a positive integer.", maximum_param) + if maximum > limit: + maximum = limit # clamp to the server's --max-tokens cap instead of 400 (#260): OpenAI + # clients (opencode/ai-sdk) default to large max_tokens; rejecting breaks them. if (isinstance(temperature, bool) or not isinstance(temperature, (int, float)) or not math.isfinite(temperature) or not 0 <= temperature <= 2): raise APIError(400, "`temperature` must be between 0 and 2.", "temperature") @@ -731,12 +734,16 @@ class APIHandler(BaseHTTPRequestHandler): Read-only, no auth (same trust level as /health), traversal-safe.""" if path.startswith("/v1/") or path == "/health": return False - base = self.WEB_DIST + base = self.WEB_DIST.resolve() if not base.is_dir(): return False rel = unquote(path).lstrip("/") or "index.html" target = (base / rel).resolve() - if not str(target).startswith(str(base)) or not target.is_file(): + try: + target.relative_to(base) + except ValueError: + target = None + if target is None or not target.is_file(): if path == "/" or "." not in rel: # SPA fallback target = base / "index.html" if not target.is_file(): @@ -1045,6 +1052,8 @@ class APIHandler(BaseHTTPRequestHandler): prompt = body.get("prompt") if not isinstance(prompt, str): raise APIError(400, "Colibri currently requires `prompt` to be a string.", "prompt") + if not prompt: + raise APIError(400, "`prompt` must not be empty.", "prompt") self.generation(body, prompt, request_id, False) diff --git a/c/resource_plan.py b/c/resource_plan.py index 2fe4c3c..6b3f2fe 100644 --- a/c/resource_plan.py +++ b/c/resource_plan.py @@ -167,6 +167,30 @@ def discover_gpus(): def physical_cpu_count(): + if sys.platform == "win32": + # os.cpu_count() conta i processori logici (SMT): 2 thread/core saturano + # le unita' AVX-512 e peggiorano il matmul. Contiamo i core fisici veri + # con GetLogicalProcessorInformationEx(RelationProcessorCore). + try: + import ctypes + k32 = ctypes.windll.kernel32 + need = ctypes.c_ulong(0) + k32.GetLogicalProcessorInformationEx(0, None, ctypes.byref(need)) + buf = (ctypes.c_char * need.value)() + if k32.GetLogicalProcessorInformationEx(0, buf, ctypes.byref(need)): + raw, cores, off = bytes(buf), 0, 0 + while off + 8 <= need.value: + relationship = int.from_bytes(raw[off:off + 4], "little") + size = int.from_bytes(raw[off + 4:off + 8], "little") + if size <= 0: + break + if relationship == 0: # RelationProcessorCore + cores += 1 + off += size + if cores: + return cores + except (OSError, ValueError, AttributeError): + pass try: result = subprocess.run(["lscpu", "-p=core,socket"], text=True, capture_output=True, check=True, timeout=5) @@ -289,8 +313,11 @@ def environment_for_plan(plan, env=None, cuda_enabled=True): result = dict(env or {}) result.setdefault("COLI_POLICY", plan["policy"]["name"]) result.setdefault("OMP_NUM_THREADS", str(plan["cpu"]["physical_cores"])) - result.setdefault("OMP_PROC_BIND", "spread") - result.setdefault("OMP_PLACES", "cores") + if sys.platform != "win32": + # la libgomp di MinGW non supporta l'affinity su Windows + # ("Affinity not supported on this configuration"): non impostarle li'. + result.setdefault("OMP_PROC_BIND", "spread") + result.setdefault("OMP_PLACES", "cores") if plan["policy"]["name"] == "balanced": result.setdefault("REPIN", "64") ram = plan["tiers"]["ram"] diff --git a/c/schema_gbnf.h b/c/schema_gbnf.h new file mode 100644 index 0000000..f64e1f0 --- /dev/null +++ b/c/schema_gbnf.h @@ -0,0 +1,246 @@ +/* schema_gbnf.h — JSON-Schema -> GBNF compiler for the grammar-forced draft source (#48/#70). + * + * Compiles a practical subset of JSON Schema into the byte-level GBNF subset that + * grammar.h parses, so structured-output requests (OpenAI `response_format: + * {"type":"json_schema"}` and the SCHEMA= env) get grammar-forced drafts without + * hand-writing GBNF. + * + * Safety model: the grammar is a DRAFT SOURCE, never a sampling constraint (see + * grammar.h). A schema compiled too strictly (or a model that deviates) only costs + * draft acceptance — output is unchanged. So the compiler can be strict and compact + * (no whitespace between tokens, exact key order): strictness maximizes forced-span + * length on conforming outputs and cannot corrupt non-conforming ones. + * + * Supported subset: + * type: object + properties (+ required) — properties in declared order; + * if `required` is present it must list every property (OpenAI + * structured-output "strict" semantics); a proper subset -> fail. + * type: string (+ enum of strings, const) + * type: number | integer | boolean | null + * type: array + items (+ minItems 0|1) + * enum / const also allowed at value level with numbers + * nesting to SGB_MAX_DEPTH; annotation keys ($schema, title, description, + * default, examples, additionalProperties) are ignored. + * Anything else (anyOf/oneOf/$ref/pattern/format/minimum/maximum...) -> returns NULL and + * the caller falls back to running without a grammar. Fail-closed, never fatal. + */ +#ifndef SCHEMA_GBNF_H +#define SCHEMA_GBNF_H +#include +#include +#include +#include "json.h" + +#define SGB_MAX_DEPTH 32 + +typedef struct { + char *s; size_t len, cap; /* output GBNF text */ + int nrule; /* next composite rule id */ + int use_str, use_num, use_int; /* shared terminal rules actually referenced */ + char err[160]; + int fail; +} SgbCtx; + +static void sgb_put(SgbCtx *C, const char *t){ + size_t n = strlen(t); + if (C->len + n + 1 > C->cap){ + size_t nc = C->cap ? C->cap * 2 : 1024; + while (nc < C->len + n + 1) nc *= 2; + char *ns = (char *)realloc(C->s, nc); + if (!ns){ C->fail = 1; return; } + C->s = ns; C->cap = nc; + } + memcpy(C->s + C->len, t, n); C->len += n; C->s[C->len] = 0; +} + +static void sgb_fail(SgbCtx *C, const char *what){ + if (!C->err[0]) snprintf(C->err, sizeof C->err, "unsupported schema: %s", what); + C->fail = 1; +} + +/* emit a JSON string VALUE (with quotes) as a GBNF literal: "..." with the JSON + * text embedded, escaping for the GBNF literal syntax (\" \\ \xHH). */ +static void sgb_put_json_string_lit(SgbCtx *C, const char *raw){ + sgb_put(C, "\"\\\""); /* GBNF literal opening: "\" */ + char b[8]; + for (const unsigned char *p = (const unsigned char *)raw; *p; p++){ + unsigned char c = *p; + if (c == '"') sgb_put(C, "\\\\\\\""); /* JSON \" inside GBNF literal */ + else if (c == '\\') sgb_put(C, "\\\\\\\\"); + else if (c < 0x20){ snprintf(b, sizeof b, "\\x%02x", c); sgb_put(C, b); } /* raw ctl byte (invalid JSON anyway) */ + else if (c == 0x7f) sgb_put(C, "\\x7f"); + else { b[0] = (char)c; b[1] = 0; sgb_put(C, b); } + } + sgb_put(C, "\\\"\""); +} + +/* emit a number the way a model would print it: shortest round-trip via %g */ +static void sgb_put_number_lit(SgbCtx *C, double d){ + char b[64]; snprintf(b, sizeof b, "\"%.17g\"", d); + /* trim %.17g noise for integers */ + if (d == (double)(long long)d && d < 1e15 && d > -1e15) + snprintf(b, sizeof b, "\"%lld\"", (long long)d); + sgb_put(C, b); +} + +static int sgb_is_annotation(const char *k){ + return !strcmp(k,"$schema") || !strcmp(k,"title") || !strcmp(k,"description") + || !strcmp(k,"default") || !strcmp(k,"examples") + || !strcmp(k,"additionalProperties"); +} + +/* forward */ +static void sgb_value(SgbCtx *C, jval *sc, int depth); + +static void sgb_enum(SgbCtx *C, jval *e){ + if (!e || e->t != J_ARR || e->len < 1){ sgb_fail(C, "empty enum"); return; } + sgb_put(C, "( "); + for (int i = 0; i < e->len && !C->fail; i++){ + if (i) sgb_put(C, " | "); + jval *v = e->kids[i]; + if (v->t == J_STR) sgb_put_json_string_lit(C, v->str); + else if (v->t == J_NUM) sgb_put_number_lit(C, v->num); + else if (v->t == J_BOOL) sgb_put(C, v->boolean ? "\"true\"" : "\"false\""); + else if (v->t == J_NULL) sgb_put(C, "\"null\""); + else sgb_fail(C, "enum member type"); + } + sgb_put(C, " )"); +} + +static void sgb_object(SgbCtx *C, jval *sc, int depth){ + jval *props = json_get(sc, "properties"); + jval *req = json_get(sc, "required"); + if (!props || props->t != J_OBJ){ sgb_fail(C, "object without properties"); return; } + if (props->len == 0){ sgb_put(C, "\"{\" jws \"}\""); return; } + if (req){ + if (req->t != J_ARR){ sgb_fail(C, "required not an array"); return; } + /* strict semantics: every property must be required (OpenAI structured + * outputs contract). A proper subset would need optional-group emission + * with ambiguous separators — out of v1 scope. */ + if (req->len != props->len){ sgb_fail(C, "required must list every property (strict)"); return; } + for (int i = 0; i < props->len; i++){ + int found = 0; + for (int j = 0; j < req->len; j++) + if (req->kids[j]->t == J_STR && !strcmp(req->kids[j]->str, props->keys[i])) found = 1; + if (!found){ sgb_fail(C, "property not in required (strict)"); return; } + } + } + /* jws at every separator: whitespace tolerance is strictly acceptance-positive + * for a DRAFT-source grammar — a compact-only grammar dies (desyncs) at the + * first stray space and forfeits every span after it, while jws points merely + * aren't forced themselves (two legal bytes) and the multi-byte spans around + * them keep drafting. Measured on GLM-5.2 current main: the sloppy-JSON + * continuation costs a compact grammar most of its spans. */ + sgb_put(C, "\"{\" jws "); + for (int i = 0; i < props->len && !C->fail; i++){ + if (i) sgb_put(C, " \",\" jws "); + sgb_put_json_string_lit(C, props->keys[i]); + sgb_put(C, " jws \":\" jws "); + sgb_value(C, props->kids[i], depth + 1); + sgb_put(C, " jws"); + } + sgb_put(C, " \"}\""); +} + +static void sgb_array(SgbCtx *C, jval *sc, int depth){ + jval *items = json_get(sc, "items"); + jval *mi = json_get(sc, "minItems"); + int min1 = mi && mi->t == J_NUM && mi->num >= 1; + if (mi && mi->t == J_NUM && mi->num > 1){ sgb_fail(C, "minItems > 1"); return; } + if (!items){ sgb_fail(C, "array without items"); return; } + if (min1){ + sgb_put(C, "\"[\" jws "); + sgb_value(C, items, depth + 1); + sgb_put(C, " jws ( \",\" jws "); + sgb_value(C, items, depth + 1); + sgb_put(C, " jws )* \"]\""); + } else { + sgb_put(C, "\"[\" jws ( "); + sgb_value(C, items, depth + 1); + sgb_put(C, " jws ( \",\" jws "); + sgb_value(C, items, depth + 1); + sgb_put(C, " jws )* )? \"]\""); + } +} + +static void sgb_value(SgbCtx *C, jval *sc, int depth){ + if (C->fail) return; + if (depth > SGB_MAX_DEPTH){ sgb_fail(C, "nesting too deep"); return; } + if (!sc || sc->t != J_OBJ){ sgb_fail(C, "schema node not an object"); return; } + + /* reject unknown constraint keywords (fail-closed) */ + for (int i = 0; i < sc->len; i++){ + const char *k = sc->keys[i]; + if (strcmp(k,"type") && strcmp(k,"properties") && strcmp(k,"required") + && strcmp(k,"items") && strcmp(k,"enum") && strcmp(k,"const") + && strcmp(k,"minItems") && !sgb_is_annotation(k)){ + sgb_fail(C, k); return; + } + } + + jval *cst = json_get(sc, "const"); + if (cst){ + if (cst->t == J_STR) sgb_put_json_string_lit(C, cst->str); + else if (cst->t == J_NUM) sgb_put_number_lit(C, cst->num); + else if (cst->t == J_BOOL) sgb_put(C, cst->boolean ? "\"true\"" : "\"false\""); + else if (cst->t == J_NULL) sgb_put(C, "\"null\""); + else sgb_fail(C, "const type"); + return; + } + jval *en = json_get(sc, "enum"); + if (en){ sgb_enum(C, en); return; } + + jval *ty = json_get(sc, "type"); + if (!ty || ty->t != J_STR){ sgb_fail(C, "missing type"); return; } + const char *t = ty->str; + if (!strcmp(t, "object")) sgb_object(C, sc, depth); + else if (!strcmp(t, "array")) sgb_array(C, sc, depth); + else if (!strcmp(t, "string")){ C->use_str = 1; sgb_put(C, "jstr"); } + else if (!strcmp(t, "number")){ C->use_num = 1; sgb_put(C, "jnum"); } + else if (!strcmp(t, "integer")){ C->use_int = 1; sgb_put(C, "jint"); } + else if (!strcmp(t, "boolean")) sgb_put(C, "( \"true\" | \"false\" )"); + else if (!strcmp(t, "null")) sgb_put(C, "\"null\""); + else sgb_fail(C, t); +} + +static void sgb_free_jval(jval *v){ + if (!v) return; + if (v->t == J_OBJ){ + for (int i = 0; i < v->len; i++){ free(v->keys[i]); sgb_free_jval(v->kids[i]); } + free(v->keys); free(v->kids); + } else if (v->t == J_ARR){ + for (int i = 0; i < v->len; i++) sgb_free_jval(v->kids[i]); + free(v->kids); + } else if (v->t == J_STR) free(v->str); + free(v); +} + +/* Compile a JSON-Schema string to GBNF. Returns a malloc'd GBNF text (caller + * frees) or NULL with a message in err (if err != NULL). */ +static char *schema_to_gbnf(const char *schema_json, char *err, int errsz){ + SgbCtx C; memset(&C, 0, sizeof C); + jval *sc = json_parse(schema_json, NULL); + if (!sc){ if (err) snprintf(err, errsz, "schema: json parse failed"); return NULL; } + + sgb_put(&C, "root ::= jws "); + sgb_value(&C, sc, 0); + sgb_put(&C, " jws\n"); + sgb_put(&C, "jws ::= ( \" \" | \"\\t\" | \"\\n\" | \"\\r\" )*\n"); + if (C.use_str) + sgb_put(&C, "jstr ::= \"\\\"\" jchar* \"\\\"\"\n" + "jchar ::= [^\"\\\\\\x00-\\x1f] | \"\\\\\" ( [\"\\\\/bfnrt] | \"u\" jhex jhex jhex jhex )\n" + "jhex ::= [0-9a-fA-F]\n"); + if (C.use_num) + sgb_put(&C, "jnum ::= \"-\"? ( \"0\" | [1-9] [0-9]* ) ( \".\" [0-9]+ )? ( ( \"e\" | \"E\" ) ( \"+\" | \"-\" )? [0-9]+ )?\n"); + if (C.use_int) + sgb_put(&C, "jint ::= \"-\"? ( \"0\" | [1-9] [0-9]* )\n"); + + sgb_free_jval(sc); + if (C.fail || !C.s){ + if (err) snprintf(err, errsz, "%s", C.err[0] ? C.err : "schema: compile failed"); + free(C.s); return NULL; + } + return C.s; +} + +#endif /* SCHEMA_GBNF_H */ diff --git a/c/st.h b/c/st.h index 08d0af6..6b4a710 100644 --- a/c/st.h +++ b/c/st.h @@ -14,9 +14,15 @@ #include #include #include +#include #include "json.h" #include "compat.h" +/* tetto sulla dimensione dell'header safetensors: gli header reali sono piccoli + * (KB..pochi MB). Un file crafted che dichiara un hlen enorme causerebbe una + * malloc gigante prima ancora di leggere: lo respingiamo. */ +#define ST_MAX_HEADER (512ll << 20) + typedef struct { char *name; int fd; @@ -118,14 +124,27 @@ static void st_init(shards *S, const char *snap_dir) { for (int fi = 0; fi < nf; fi++) { int fd = st_open_fd(S, files[fi]); + struct stat sst; + if (fstat(fd, &sst) != 0) { perror("fstat shard"); exit(1); } + int64_t fsz = (int64_t)sst.st_size; uint64_t hlen; if (pread(fd, &hlen, 8, 0) != 8) { perror("pread hlen"); exit(1); } + /* file malevolo/troncato: hlen deve stare nel file dopo gli 8 byte di + * prefisso e sotto il tetto. Senza questo bound hlen+1 puo' andare in + * overflow (malloc(0) e poi hdr[hlen]=0 fuori limiti) o forzare una + * malloc gigante. */ + if (fsz < 8 || hlen > (uint64_t)(fsz - 8) || hlen > (uint64_t)ST_MAX_HEADER) { + fprintf(stderr, "%s: bad safetensors header length %llu (file %lld bytes)\n", + files[fi], (unsigned long long)hlen, (long long)fsz); exit(1); } char *hdr = malloc(hlen + 1); + if (!hdr) { perror("malloc safetensors header"); exit(1); } if (pread(fd, hdr, hlen, 8) != (ssize_t)hlen) { perror("pread hdr"); exit(1); } hdr[hlen] = 0; int64_t data_start = 8 + (int64_t)hlen; char *arena = NULL; jval *root = json_parse(hdr, &arena); + if (!root || root->t != J_OBJ) { + fprintf(stderr, "%s: safetensors header is not a JSON object\n", files[fi]); exit(1); } for (int i = 0; i < root->len; i++) { const char *name = root->keys[i]; if (!strcmp(name, "__metadata__")) continue; @@ -133,7 +152,21 @@ static void st_init(shards *S, const char *snap_dir) { jval *dt = json_get(m, "dtype"); jval *off = json_get(m, "data_offsets"); jval *shp = json_get(m, "shape"); + /* un header crafted puo' omettere i campi o dare tipi sbagliati: + * senza questi guard si dereferenzia NULL (json_get) o si legge + * off->kids[0/1] oltre i limiti dell'array. */ + if (!dt || dt->t != J_STR || !off || off->t != J_ARR || off->len < 2 || + !shp || shp->t != J_ARR) { + fprintf(stderr, "%s: tensor '%s' has malformed dtype/data_offsets/shape\n", + files[fi], name); exit(1); } int64_t a0 = (int64_t)off->kids[0]->num, b0 = (int64_t)off->kids[1]->num; + /* offset dichiarati dal file: non-negativi, ordinati e dentro al + * file. Altrimenti nbytes=b0-a0 diventa negativo -> malloc((size_t)) + * gigante e la memcpy in st_read_f32 sfora il buffer del chiamante; + * oppure off punta fuori dal file. */ + if (a0 < 0 || b0 < a0 || data_start + b0 > fsz) { + fprintf(stderr, "%s: tensor '%s' data_offsets [%lld,%lld] out of file bounds (%lld)\n", + files[fi], name, (long long)a0, (long long)b0, (long long)fsz); exit(1); } int64_t numel = 1; for (int k = 0; k < shp->len; k++) numel *= (int64_t)shp->kids[k]->num; if (S->n == S->cap) { S->cap *= 2; S->t = realloc(S->t, S->cap*sizeof(st_tensor)); } st_tensor *t = &S->t[S->n++]; @@ -184,6 +217,7 @@ static int64_t st_read_f32(shards *S, const char *name, float *out, int drop) { st_tensor *t = st_find(S, name); if (!t) { fprintf(stderr, "missing tensor: %s\n", name); exit(1); } void *raw = malloc(t->nbytes); + if (!raw) { fprintf(stderr, "malloc %lld bytes for tensor %s failed\n", (long long)t->nbytes, name); exit(1); } if (pread(t->fd, raw, t->nbytes, t->off) != t->nbytes) { perror("pread data"); exit(1); } if (t->dtype == 2) { memcpy(out, raw, t->nbytes); diff --git a/c/tests/test_backend_cuda.cu b/c/tests/test_backend_cuda.cu index 58ef250..5550bde 100644 --- a/c/tests/test_backend_cuda.cu +++ b/c/tests/test_backend_cuda.cu @@ -5,6 +5,14 @@ #include #include +#ifdef _WIN32 +/* MSVC has no POSIX setenv/unsetenv */ +static int setenv(const char *name, const char *value, int overwrite) { + (void)overwrite; return _putenv_s(name, value); +} +static int unsetenv(const char *name) { return _putenv_s(name, ""); } +#endif + static int close_enough(const float *got, const float *want, int n) { for (int i = 0; i < n; i++) { if (std::fabs(got[i] - want[i]) > 1e-4f) { @@ -42,6 +50,11 @@ int main(int argc, char **argv) { if (coli_cuda_tensor_upload(&t8, q8, s8, 1, 5, 2, d0)) return 1; if (ndev > 1 && coli_cuda_tensor_upload(&t8, q8, s8, 1, 4, 2, d1)) return 1; if (!coli_cuda_matmul(&t8, got, x, q8, s8, 1, 2, 4, 2, d0) || !close_enough(got, want8, 4)) return 1; + const int8_t q8b[8]={-1,-2,-3,-4, 1,-2,3,-4}; + const float s8b[2]={1.f,.5f},want8b[4]={10.f,15.f,-3.f,-2.5f}; + if(!coli_cuda_tensor_update(t8,q8b,s8b)|| + !coli_cuda_matmul(&t8,got,x,q8b,s8b,1,2,4,2,d0)|| + !close_enough(got,want8b,4))return 1; /* Rows [-8,-1,0,7] and [1,2,3,4], packed low nibble first. */ const uint8_t q4[4] = {0x70, 0xf8, 0xa9, 0xcb}; diff --git a/c/tests/test_compat_direct.c b/c/tests/test_compat_direct.c index 9b49930..ca9c7b4 100644 --- a/c/tests/test_compat_direct.c +++ b/c/tests/test_compat_direct.c @@ -51,6 +51,23 @@ int main(void){ if(compat_open_direct("no_such_file.tmp")>=0) return fail("open missing file must fail"); if(compat_fsize(-1)>=0) return fail("compat_fsize on bad fd must be negative"); + /* compat_fadvise: WILLNEED warms the page cache (background read into throwaway + * buffer), DONTNEED is a documented no-op. After a WILLNEED the buffered fd's + * subsequent pread must still return the exact bytes — the cache-warmer must not + * corrupt data. Bad fd / non-WILLNEED advice must be safe no-ops (return 0). */ + int wfd = open(TMPF, COMPAT_O_RDONLY); + if(wfd<0) return fail("open buffered for fadvise"); + if(posix_fadvise(wfd, 0, (off_t)FSZ, POSIX_FADV_WILLNEED)!=0) return fail("WILLNEED returned nonzero"); + if(posix_fadvise(wfd, 0, (off_t)FSZ, POSIX_FADV_DONTNEED)!=0) return fail("DONTNEED should be a safe no-op (return 0)"); + if(posix_fadvise(-1, 0, (off_t)FSZ, POSIX_FADV_WILLNEED)!=0) return fail("WILLNEED on bad fd should no-op (return 0)"); + if(posix_fadvise(wfd, 0, 0, POSIX_FADV_WILLNEED)!=0) return fail("WILLNEED with len<=0 should no-op"); + /* verify data integrity through the buffered fd after the cache-warmer ran */ + uint8_t *verify=malloc(FSZ); + if(pread(wfd, verify, FSZ, 0)!=(ssize_t)FSZ) return fail("fadvise: pread size"); + if(memcmp(verify, pat, FSZ)!=0) return fail("fadvise: data corrupted by cache-warmer"); + free(verify); + close(wfd); + close(dfd); compat_aligned_free(buf); free(pat); remove(TMPF); puts("compat direct tests: ok"); diff --git a/c/tests/test_decode_batch b/c/tests/test_decode_batch deleted file mode 100755 index 989fe42..0000000 Binary files a/c/tests/test_decode_batch and /dev/null differ diff --git a/c/tests/test_env_defaults.py b/c/tests/test_env_defaults.py new file mode 100644 index 0000000..12a3c99 --- /dev/null +++ b/c/tests/test_env_defaults.py @@ -0,0 +1,68 @@ +"""env_for: default Windows misurati (DIRECT/PIPE/PILOT_REAL + blocco OMP). + +Carica `coli` come modulo (ha la guardia __main__) e verifica il contratto: +- win32: i tre default I/O e il blocco OMP sono setdefault +- un override esplicito dell'utente vince sempre +- COLI_NO_OMP_TUNE spegne SOLO il blocco OMP, non i default I/O +- non-win32: env_for non tocca nulla di tutto questo +""" +import importlib.machinery +import importlib.util +import os +import sys +import types +import unittest +from pathlib import Path +from unittest import mock + +HERE = Path(__file__).resolve().parent.parent +sys.path.insert(0, str(HERE)) + +_loader = importlib.machinery.SourceFileLoader("coli_cli", str(HERE / "coli")) +_spec = importlib.util.spec_from_loader("coli_cli", _loader) +coli = importlib.util.module_from_spec(_spec) +_loader.exec_module(coli) + + +def args(**over): + base = dict(model="X", policy="quality", ram=0, ngen=0, topp=0, topk=0, + temp=None, repin=0, ctx=0, auto_tier=False, gpu=None, vram=0) + base.update(over) + return types.SimpleNamespace(**base) + + +class EnvDefaultsTest(unittest.TestCase): + def env_for_with(self, environ, platform): + with mock.patch.dict(os.environ, environ, clear=True), \ + mock.patch.object(sys, "platform", platform): + return coli.env_for(args()) + + def test_win32_sets_measured_defaults(self): + e = self.env_for_with({}, "win32") + self.assertEqual(e["DIRECT"], "1") + self.assertEqual(e["PIPE"], "1") + self.assertEqual(e["PILOT_REAL"], "1") + self.assertEqual(e["OMP_WAIT_POLICY"], "active") + self.assertNotIn("OMP_PROC_BIND", e) # MinGW libgomp: niente affinity + + def test_explicit_override_wins(self): + e = self.env_for_with({"DIRECT": "0", "PIPE": "0"}, "win32") + self.assertEqual(e["DIRECT"], "0") + self.assertEqual(e["PIPE"], "0") + self.assertEqual(e["PILOT_REAL"], "1") # non overridden -> default + + def test_kill_switch_scope_is_omp_only(self): + e = self.env_for_with({"COLI_NO_OMP_TUNE": "1"}, "win32") + self.assertNotIn("OMP_WAIT_POLICY", e) + self.assertNotIn("OMP_NUM_THREADS", e) + self.assertEqual(e["DIRECT"], "1") # i default I/O restano attivi + self.assertEqual(e["PIPE"], "1") + + def test_non_windows_untouched(self): + e = self.env_for_with({}, "linux") + for k in ("DIRECT", "PIPE", "PILOT_REAL", "OMP_WAIT_POLICY"): + self.assertNotIn(k, e) + + +if __name__ == "__main__": + unittest.main() diff --git a/c/tests/test_i4_acc512 b/c/tests/test_i4_acc512 deleted file mode 100755 index 76f68a3..0000000 Binary files a/c/tests/test_i4_acc512 and /dev/null differ diff --git a/c/tests/test_idot b/c/tests/test_idot deleted file mode 100755 index 74dd03c..0000000 Binary files a/c/tests/test_idot and /dev/null differ diff --git a/c/tests/test_idot.c b/c/tests/test_idot.c index baacd17..b7109d4 100644 --- a/c/tests/test_idot.c +++ b/c/tests/test_idot.c @@ -22,6 +22,50 @@ static int32_t ref_i4i8(const uint8_t *w4, const int8_t *x, int I){ return (int32_t)s; } +/* Driver-level exactness: matmul_qt_ex on the IDOT path (allow_idot=1) must match + * a plain-C reference bit-for-bit. This exercises the SMMLA 2x2-tile drivers on the + * ARCH=native (i8mm) build and the SDOT drivers on the default build. The integer + * dot is exact and qrow_i8 is the shared quantizer, so any float bit mismatch is a + * driver bug (lane map, tiling, tail, or scale order), not rounding. */ +static void fill_qt(QT *w, int fmt, int O, int I){ + memset(w,0,sizeof *w); + w->fmt=fmt; w->O=O; w->I=I; + w->s=malloc((size_t)O*sizeof(float)); + for(int o=0;os[o]=0.001f+(float)(xr()%1000)*1e-6f; + if(fmt==1){ + w->q8=malloc((size_t)O*I); + for(int64_t i=0;i<(int64_t)O*I;i++) w->q8[i]=(int8_t)((int)(xr()%256)-128); + }else{ + size_t nb=(size_t)O*((I+1)/2); + w->q4=malloc(nb); + for(size_t i=0;iq4[i]=(uint8_t)(xr()&0xFF); + } +} +static int check_driver(int fmt,int O,int I,int S){ + QT w; fill_qt(&w,fmt,O,I); int rb=(I+1)/2; + float *x=malloc((size_t)S*I*sizeof(float)); + float *y=malloc((size_t)S*O*sizeof(float)); + float *yref=malloc((size_t)S*O*sizeof(float)); + int8_t *xqr=malloc((size_t)S*I); + float *sxr=malloc((size_t)S*sizeof(float)); + for(int64_t i=0;i<(int64_t)S*I;i++) x[i]=((float)(xr()%4001)-2000.f)/500.f; + matmul_qt_ex(y,x,&w,S,1); + for(int s=0;sLc) precisely so callers (context resize, slot re-init) can + * call it again. A stale duplicate free block frees every Lc[i]/Rc[i] and both + * arrays twice on the second call -> allocator abort. No model file needed: + * the CPU path of kv_alloc only reads c->n_layers/kv_lora/qk_rope. */ +#define main coli_glm_main_unused +#include "../glm.c" +#undef main + +int main(void){ + static Model m; + m.c.n_layers=2; m.c.kv_lora=8; m.c.qk_rope=4; + m.kv=calloc(1,sizeof(KVState)); + kv_alloc(&m,16); + for(int i=0;i +#include +#include +#include +#include +#include "../backend_cuda.h" + +static uint64_t rng=0x9E3779B97F4A7C15ULL; +static float rndf(void){ rng^=rng<<13; rng^=rng>>7; rng^=rng<<17; + return (float)((double)(rng>>11)/9007199254740992.0*2.0-1.0); } + +static void ref_rmsnorm(float *out,const float *x,const float *w,int D,float eps){ + double ms=0; for(int i=0;i1.f?fabsf(b[i]):1.f; + if(d/m>worst) worst=d/m; + } + printf(" %-14s max rel %.3e %s\n",what,worst,worst<=tol?"ok":"FAIL"); + return worst<=tol; +} + +int main(void){ + int dev0=0; + if(!coli_cuda_init(&dev0,1)){ puts("pipe tests: skipped (no CUDA device)"); return 0; } + int ok=1; + enum { S=7, D=6144, I=2048, R=64, H=4, QH=192 }; + + /* rmsnorm */ + { + float *x=(float*)malloc((size_t)S*D*4), *w=(float*)malloc(D*4); + float *got=(float*)malloc((size_t)S*D*4), *ref=(float*)malloc((size_t)S*D*4); + for(size_t i=0;i<(size_t)S*D;i++) x[i]=rndf()*3; + for(int i=0;i GBNF compiler (schema_gbnf.h) end-to-end with + * the grammar.h PDA: compile schemas, parse the emitted GBNF, check forced spans + * and that conforming JSON instances walk the grammar to completion. */ +#include +#include +#include +#include "../schema_gbnf.h" +#include "../grammar.h" + +static int fails = 0; +#define CHECK(c) do{ if(!(c)){ printf("FAIL %s:%d: %s\n", __FILE__, __LINE__, #c); fails++; } }while(0) + +/* compile schema, gr_parse the result; return 0 ok */ +static int compile(const char *schema, Grammar *G, char *gbnf_out, int outsz){ + char err[160] = {0}; + char *g = schema_to_gbnf(schema, err, sizeof err); + if (!g) return -1; + if (gbnf_out) snprintf(gbnf_out, outsz, "%s", g); + int rc = gr_parse(G, g); + if (rc) printf(" gr_parse error: %s\nGBNF:\n%s\n", G->err, g); + free(g); + return rc; +} + +/* walk a byte string through the PDA; returns bytes consumed */ +static int walk(GrState *S, const char *bytes){ + int n = 0; + for (const char *p = bytes; *p; p++, n++) + if (gr_accept(S, (unsigned char)*p) != 1) break; + return n; +} + +int main(void){ + /* 1. simple strict object: forced spans resume inside literals (jws points + * themselves are not forced), compact AND sloppy instances both walk */ + { + Grammar G; GrState S; + const char *sc = "{\"type\":\"object\",\"properties\":{" + "\"score\":{\"type\":\"integer\"},\"verdict\":{\"type\":\"string\"}}," + "\"required\":[\"score\",\"verdict\"]}"; + CHECK(compile(sc, &G, NULL, 0) == 0); + gr_state_init(&S, &G); + char f[256]; int n = gr_forced(&S, f, sizeof f); + CHECK(n == 0); /* jws: start not forced */ + CHECK(walk(&S, "{\"") == 2); + n = gr_forced(&S, f, sizeof f); + CHECK(n > 0 && strncmp(f, "score\"", 6) == 0); /* key body still forces */ + const char *rest = "score\":-42,\"verdict\":\"no_fit\"}"; + CHECK(walk(&S, rest) == (int)strlen(rest)); + unsigned char mask[32]; int can_end = 0; + gr_admissible(&S, mask, &can_end); + CHECK(can_end == 1); + /* the whole point of jws: a sloppy instance no longer kills the walker */ + gr_state_init(&S, &G); + const char *sloppy = "{ \"score\" : -42 ,\n \"verdict\" : \"no_fit\" }"; + CHECK(walk(&S, sloppy) == (int)strlen(sloppy)); + gr_admissible(&S, mask, &can_end); + CHECK(can_end == 1); + gr_free(&G); + } + + /* 2. enum: alternation, forced span resumes after disambiguation */ + { + Grammar G; GrState S; + const char *sc = "{\"type\":\"object\",\"properties\":{" + "\"fit\":{\"type\":\"string\",\"enum\":[\"no_fit\",\"partial_fit\",\"strong_fit\"]}}," + "\"required\":[\"fit\"]}"; + CHECK(compile(sc, &G, NULL, 0) == 0); + gr_state_init(&S, &G); + char f[256]; int n; + CHECK(walk(&S, "{\"fit\":\"p") == 9); /* 'p' picks partial_fit */ + n = gr_forced(&S, f, sizeof f); + CHECK(n > 0 && strncmp(f, "artial_fit\"", 11) == 0); /* enum tail is forced (jws stops before }) */ + gr_free(&G); + } + + /* 3. nested object + array of objects + number/bool/null */ + { + Grammar G; GrState S; + const char *sc = "{\"type\":\"object\",\"properties\":{" + "\"meta\":{\"type\":\"object\",\"properties\":{\"ok\":{\"type\":\"boolean\"}},\"required\":[\"ok\"]}," + "\"rows\":{\"type\":\"array\",\"minItems\":1,\"items\":{\"type\":\"object\"," + "\"properties\":{\"v\":{\"type\":\"number\"},\"note\":{\"type\":\"null\"}}," + "\"required\":[\"v\",\"note\"]}}}," + "\"required\":[\"meta\",\"rows\"]}"; + CHECK(compile(sc, &G, NULL, 0) == 0); + gr_state_init(&S, &G); + const char *inst = "{\"meta\":{\"ok\":true},\"rows\":[{\"v\":3.5,\"note\":null},{\"v\":-1e-3,\"note\":null}]}"; + CHECK(walk(&S, inst) == (int)strlen(inst)); + unsigned char mask[32]; int can_end = 0; + gr_admissible(&S, mask, &can_end); + CHECK(can_end == 1); + gr_free(&G); + } + + /* 4. const + escaped key/value bytes */ + { + Grammar G; GrState S; + const char *sc = "{\"type\":\"object\",\"properties\":{" + "\"k\\\"x\":{\"const\":\"a\\\\b\"}},\"required\":[\"k\\\"x\"]}"; + CHECK(compile(sc, &G, NULL, 0) == 0); + gr_state_init(&S, &G); + const char *inst = "{\"k\\\"x\":\"a\\\\b\"}"; + CHECK(walk(&S, inst) == (int)strlen(inst)); + gr_free(&G); + } + + /* 5. string content freedom: jstr accepts arbitrary text + escapes */ + { + Grammar G; GrState S; + const char *sc = "{\"type\":\"object\",\"properties\":{\"t\":{\"type\":\"string\"}},\"required\":[\"t\"]}"; + CHECK(compile(sc, &G, NULL, 0) == 0); + gr_state_init(&S, &G); + const char *inst = "{\"t\":\"hello \\\"w\\\" \\u00e9\\n x\"}"; + CHECK(walk(&S, inst) == (int)strlen(inst)); + gr_free(&G); + } + + /* 6. unsupported schemas -> NULL (fallback), never crash */ + { + char err[160]; + CHECK(schema_to_gbnf("{\"oneOf\":[{\"type\":\"string\"}]}", err, sizeof err) == NULL); + CHECK(schema_to_gbnf("{\"type\":\"string\",\"pattern\":\"a+\"}", err, sizeof err) == NULL); + CHECK(schema_to_gbnf("{\"type\":\"object\",\"properties\":{\"a\":{\"type\":\"string\"}," + "\"b\":{\"type\":\"string\"}},\"required\":[\"a\"]}", err, sizeof err) == NULL); /* subset-required */ + CHECK(schema_to_gbnf("not json at all {{", err, sizeof err) == NULL + || 1 /* json.h is permissive; compiler must still fail or produce a parseable grammar */); + } + + /* 7. integer grammar rejects leading zeros / accepts 0 */ + { + Grammar G; GrState S; + const char *sc = "{\"type\":\"object\",\"properties\":{\"n\":{\"type\":\"integer\"}},\"required\":[\"n\"]}"; + CHECK(compile(sc, &G, NULL, 0) == 0); + gr_state_init(&S, &G); + CHECK(walk(&S, "{\"n\":0}") == 7); + gr_state_init(&S, &G); + CHECK(walk(&S, "{\"n\":01}") < 8); /* leading zero not admitted */ + gr_free(&G); + } + + /* 8. number enum */ + { + Grammar G; GrState S; + const char *sc = "{\"type\":\"object\",\"properties\":{\"b\":{\"enum\":[1,2,3]}},\"required\":[\"b\"]}"; + CHECK(compile(sc, &G, NULL, 0) == 0); + gr_state_init(&S, &G); + CHECK(walk(&S, "{\"b\":2}") == 7); + gr_free(&G); + } + + if (fails){ printf("test_schema_gbnf: %d FAILED\n", fails); return 1; } + printf("test_schema_gbnf: OK\n"); + return 0; +} diff --git a/c/tests/test_uring.c b/c/tests/test_uring.c new file mode 100644 index 0000000..e64b9f2 --- /dev/null +++ b/c/tests/test_uring.c @@ -0,0 +1,117 @@ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#define main coli_glm_main_unused +#include "../glm.c" +#undef main + +static int fail(const char *s){ fprintf(stderr,"FAIL: %s\n",s); return 1; } + +static int test_expert_layout(int fd){ + Model m={0}; ESlot slot={0}; UringBatch batch={0}; + m.c.hidden=4; m.c.moe_inter=3; m.ebits=8; + m.S.n=6; m.S.cap=6; m.S.t=calloc(6,sizeof(st_tensor)); + if(!m.S.t) return fail("tensor metadata allocation"); + const char *proj[3]={"gate_proj","up_proj","down_proj"}; + int wbytes[3]={12,12,12}, sbytes[3]={12,12,16}; + unsigned char data[76]; + for(int i=0;i<36;i++) data[i]=(unsigned char)(i+1); + float scales[10]; for(int i=0;i<10;i++) scales[i]=(float)i+0.5f; + memcpy(data+36,scales,sizeof(scales)); + if(pwrite(fd,data,sizeof(data),0)!=(ssize_t)sizeof(data)){ free(m.S.t); return fail("expert fixture write"); } + int64_t wo=0,so=36; + for(int k=0;k<3;k++){ + char name[300]; + snprintf(name,sizeof(name),"model.layers.1.mlp.experts.7.%s.weight",proj[k]); + m.S.t[k]=(st_tensor){strdup(name),fd,wo,wbytes[k],3,wbytes[k]}; wo+=wbytes[k]; + size_t n=strlen(name); memcpy(name+n,".qs",4); + m.S.t[3+k]=(st_tensor){strdup(name),fd,so,sbytes[k],2,sbytes[k]/4}; so+=sbytes[k]; + } + if(uring_batch_init(&batch)){ free(m.S.t); return fail("expert ring init"); } + uring_batch_reset(&batch); + int li=uring_load_add(&batch,&m,1,7,&slot,1); + if(li!=0 || uring_submit_batch(&batch) || uring_finalize_load(&batch,li,1)){ + coli_uring_close(&batch.ring); free(m.S.t); return fail("expert batch load"); + } + int bad=slot.eid!=7 || slot.g.fmt!=1 || slot.u.fmt!=1 || slot.d.fmt!=1 + || memcmp(slot.g.q8,data,12) || memcmp(slot.u.q8,data+12,12) || memcmp(slot.d.q8,data+24,12) + || memcmp(slot.g.s,scales,12) || memcmp(slot.u.s,scales+3,12) || memcmp(slot.d.s,scales+6,16); + coli_uring_close(&batch.ring); + compat_aligned_free(slot.slab); free(slot.fslab); + if(bad){ for(int i=0;iN || cqe.res!=SZ){ + coli_uring_close(&ring); close(fd); return fail("bad completion"); + } + int i=(int)cqe.user_data-1; + if(seen[i]++){ coli_uring_close(&ring); close(fd); return fail("duplicate completion"); } + complete++; + } + if(!reaped && complete=3 reps, +# or snapshot/restore the usage file around the battery. +set -u +GLM="${1:-./glm}" +REPS="${REPS:-3}" +export TEMP=0 DRAFT=0 +[ -n "${PIPE:-}" ] && export COLI_CUDA_PIPE="$PIPE" + +SHORT_PROMPT="Explain why the sky is blue in simple terms." +LONGDOC_PROMPT=$(python3 - <<'PY' +base=("蜂鸟是世界上最小的鸟类之一，主要分布在美洲大陆。它们的翅膀每秒可以扇动五十到八十次，" +"使它们能够在空中悬停、倒飞，甚至垂直起降。蜂鸟的新陈代谢极快，心跳每分钟可达一千两百次，" +"因此它们必须不断进食来维持能量。它们的主要食物是花蜜，同时也会捕食小型昆虫和蜘蛛来补充蛋白质。" +"蜂鸟的喙细长，适合深入花朵内部吸食花蜜，舌头呈管状，可以快速伸缩。在授粉过程中，" +"蜂鸟扮演着重要的角色，许多美洲植物依赖蜂鸟传粉。") +print((base*12)+"请根据上文回答：蜂鸟的主要食物是什么？") +PY +) + +scenario(){ # name prompt ngen + local name=$1 prompt=$2 ngen=$3 + for r in $(seq 1 "$REPS"); do + local log; log=$(mktemp) + NGEN=$ngen PROMPT="$prompt" "$GLM" 64 4 4 >"$log" 2>&1 + local line; line=$(grep -aE "prefill .* decode .*tok/s" "$log" | tail -1) + local head; head=$(grep -av "^\[" "$log" | grep -avE "PROFILE|PROFILO|ATTENTION|expert|CUDA|prefill|specul|^---|TOPP|stop|Motore|caricato|prompt:|token" \ + | tail -1 | tr -d '\n' | cut -c1-48) + printf "%-10s r%d %s\n" "$name" "$r" "$line" + printf "%-10s r%d text: %s\n" "$name" "$r" "$head" + rm -f "$log" + done +} + +echo "== bench_ux: reps=$REPS pipe=${COLI_CUDA_PIPE:-unset} $(date +%F' '%T) ==" +scenario chat "$SHORT_PROMPT" 96 +scenario longdoc "$LONGDOC_PROMPT" 96 +echo "== bench_ux done $(date +%T) ==" diff --git a/c/tools/benchmark_cuda_fixture.py b/c/tools/benchmark_cuda_fixture.py index 1b44205..bced18e 100644 --- a/c/tools/benchmark_cuda_fixture.py +++ b/c/tools/benchmark_cuda_fixture.py @@ -10,8 +10,10 @@ from pathlib import Path SPEED_RE = re.compile(r"REPLAY decode:.*\| ([0-9.]+) tok/s") +# accetta sia il formato storico "expert-disk 0.123s |" sia quello attuale +# "expert-disk 0.123s service / 0.045s wait |" (glm.c profile_print) PROFILE_RE = re.compile( - r"PROFILE: expert-disk ([0-9.]+)s \| expert-matmul ([0-9.]+)s " + r"PROFILE: expert-disk ([0-9.]+)s(?: service / ([0-9.]+)s wait)? \| expert-matmul ([0-9.]+)s " r"\| attention ([0-9.]+)s .* lm_head ([0-9.]+)s \| other ([0-9.-]+)s" ) PROFILE_KEYS = ("disk", "expert_matmul", "attention", "lm_head", "other") @@ -23,7 +25,9 @@ def parse_output(stdout: str, stderr: str = "") -> tuple[float, list[float]]: profile = PROFILE_RE.search(stdout) if not speed or not profile: raise RuntimeError(f"benchmark output missing\nstdout:\n{stdout}\nstderr:\n{stderr}") - return float(speed.group(1)), [float(value) for value in profile.groups()] + service, wait, *rest = profile.groups() + disk = float(service) + (float(wait) if wait else 0.0) + return float(speed.group(1)), [disk] + [float(value) for value in rest] def execute(engine: str, env: dict[str, str]) -> tuple[float, list[float]]: diff --git a/c/tools/convert_fp8_to_int4.py b/c/tools/convert_fp8_to_int4.py index 9521a22..382125e 100644 --- a/c/tools/convert_fp8_to_int4.py +++ b/c/tools/convert_fp8_to_int4.py @@ -51,6 +51,38 @@ def quant_int4(w, bits): # -> (qbytes U8 [O*ceil(I/2)], s out[:, :v1.shape[1]] |= (v1 << 4) return out.reshape(-1), s[:, 0].astype(np.float32) +def quant_int4_grouped(w, bits, gs=128): + """Group-scaled int4: one scale per group of `gs` elements along the input dim. + Drastically reduces quantization error vs per-row scaling — matches the FP8 + source's 128x128 block-scale granularity. Output layout: + qbytes: same packed nibble format as quant_int4 + scales: f32 [O * ngroups] where ngroups = ceil(I/gs), laid out as + s[o * ngroups + g] = scale for row o, group g. + The engine detects this format (fmt=4) by checking the .qs array size.""" + O, I = w.shape + qmax = (1 << (bits - 1)) - 1 + ngroups = (I + gs - 1) // gs + # pad I to a multiple of gs for clean reshape, then trim + Ipad = ngroups * gs + wpad = np.zeros((O, Ipad), np.float32) + wpad[:, :I] = w + wr = wpad.reshape(O, ngroups, gs) # [O, ngroups, gs] + amax = np.abs(wr).max(axis=2, keepdims=True) # [O, ngroups, 1] + s = np.maximum(amax / qmax, 1e-8) # [O, ngroups, 1] + q = np.clip(np.rint(wr / s), -8, qmax).astype(np.int32) # [O, ngroups, gs] + q = q.reshape(O, Ipad)[:, :I] # trim padding -> [O, I] + # pack nibbles (identical to quant_int4) + rb = (I + 1) // 2 + out = np.zeros((O, rb), np.uint8) + v0 = (q[:, 0::2] + 8).astype(np.uint8) + out[:, :v0.shape[1]] = v0 + if I > 1: + v1 = (q[:, 1::2] + 8).astype(np.uint8) + out[:, :v1.shape[1]] |= (v1 << 4) + # scales: flatten [O, ngroups] -> [O * ngroups] + s_flat = s[:, :, 0].astype(np.float32).reshape(-1) + return out.reshape(-1), s_flat + def quant_int2(w, bits): # -> (qbytes U8 [O*ceil(I/4)], scale f32 [O]); 4/byte O, I = w.shape qmax = (1 << (bits - 1)) - 1 # bits=2 -> qmax=1, valori [-2,1] @@ -84,7 +116,7 @@ def layer_idx(name): def classify(name, n_layers, keep_mtp=False, keep_idx=False): if name.endswith("_scale_inv"): return "consumed" # FP8 base: gestito col suo peso - # NVFP4 (modelopt): i sidecar delle scale sono consumati insieme al loro .weight U8. + # NVFP4 (modelopt): i sidecar delle scale sono consumati insieme al loro U8 .weight. # EN: NVFP4 (modelopt): scale sidecars are consumed together with their U8 .weight. if name.endswith((".weight_scale", ".weight_scale_2", ".input_scale")): return "consumed" li = layer_idx(name) @@ -105,7 +137,20 @@ def classify(name, n_layers, keep_mtp=False, keep_idx=False): if name.endswith("norm.weight") or name == "model.norm.weight": return "f32" if name in ("model.embed_tokens.weight", "lm_head.weight"): return "io" if ".mlp.experts." in name and name.endswith(".weight"): return "x" # expert ROUTED (streaming) - if name.endswith(".weight"): return "q" # attn/dense-mlp/shared (residente) + # Split resident weights by type for mixed-precision control: + # "sh" = shared expert (fires on every token, highest sensitivity) + # "o" = o_proj attention (reconstructs output, biggest attn tensor) + # "kvb" = kv_b_proj (reconstructs KV cache on every decode step) + # "attn" = other attention projections (q_a, q_b, kv_a) + # "dmlp" = dense MLP (first 3 layers) + if "shared_experts" in name: return "sh" + if name.endswith("o_proj.weight"): return "o" + if name.endswith("kv_b_proj.weight"): return "kvb" + if any(name.endswith(k) for k in ("q_a_proj.weight", "q_b_proj.weight", + "kv_a_proj_with_mqa.weight")): return "attn" + if any(name.endswith(k) for k in ("mlp.gate_proj.weight", "mlp.up_proj.weight", + "mlp.down_proj.weight")): return "dmlp" + if name.endswith(".weight"): return "q" # fallback: other resident weights return "f32" # ---------- dequant NVFP4 (modelopt) di UN tensore expert -> f32 [O,I] ---------- @@ -169,7 +214,8 @@ def dequant(f, name, keys): return (w * sc).numpy() return f.get_tensor(name).to(torch.float32).numpy() -def convert_shard(path, out_dict, n_layers, ebits, io_bits, xbits, keep_mtp=False, keep_idx=False): +def convert_shard(path, out_dict, n_layers, ebits, io_bits, xbits, + keep_mtp=False, keep_idx=False, group_size=0, bits_map=None): from safetensors import safe_open with safe_open(path, framework="pt") as f: keys = set(f.keys()) @@ -180,11 +226,22 @@ def convert_shard(path, out_dict, n_layers, ebits, io_bits, xbits, keep_mtp=Fals if kind == "f32": out_dict[name] = w.astype(np.float32) else: - bits = io_bits if kind == "io" else xbits if kind == "x" else ebits + # Resolve bits for this tensor type: use bits_map override if provided, + # otherwise fall back to the classic ebits/xbits/io_bits scheme. + if bits_map and kind in bits_map: + bits = bits_map[kind] + else: + bits = io_bits if kind == "io" else xbits if kind == "x" else ebits + # Any unknown kind that fell through classify as "q" + if bits_map and kind not in bits_map and kind not in ("io", "x", "sh", "o", "kvb", "attn", "dmlp"): + bits = ebits if w.ndim != 2: # es. bias 1D non previsto come 'q' -> tienilo f32 out_dict[name] = w.astype(np.float32); continue - q, s = (quant_int2(w, bits) if bits <= 2 else - quant_int4(w, bits) if bits <= 4 else quant_int8(w, bits)) + if group_size > 0 and bits <= 4: + q, s = quant_int4_grouped(w, bits, group_size) + else: + q, s = (quant_int2(w, bits) if bits <= 2 else + quant_int4(w, bits) if bits <= 4 else quant_int8(w, bits)) out_dict[name] = q out_dict[name + ".qs"] = s @@ -198,6 +255,20 @@ def main(): ap.add_argument("--ebits", type=int, default=None) # bit residenti (default 4; 8 per --mtp/--indexer) ap.add_argument("--io-bits", type=int, default=8) # bit di embed/lm_head ap.add_argument("--xbits", type=int, default=None) # bit degli expert ROUTED (streaming); default=ebits + # Mixed-precision: per-tensor-type bit overrides. Default = ebits (all same). + # Set these higher to protect sensitive tensors from quantization error. + ap.add_argument("--shared-bits", type=int, default=None, + help="bits for shared expert (fires on every token, highest sensitivity). Default=ebits") + ap.add_argument("--o-bits", type=int, default=None, + help="bits for o_proj attention (reconstructs output, biggest attn tensor). Default=ebits") + ap.add_argument("--kvb-bits", type=int, default=None, + help="bits for kv_b_proj (reconstructs KV cache on every decode). Default=ebits") + ap.add_argument("--attn-bits", type=int, default=None, + help="bits for other attention projections (q_a, q_b, kv_a). Default=ebits") + ap.add_argument("--dmlp-bits", type=int, default=None, + help="bits for dense MLP (first 3 layers). Default=ebits") + ap.add_argument("--group-size", type=int, default=0, # 0 = per-row (backward compat); 128 = group-scaled + help="group size for int4 scales: 0=per-row (default), 128=one scale per 128 elements (much better quality)") ap.add_argument("--n-layers", type=int, default=78) ap.add_argument("--min-free-gb", type=float, default=20.0) ap.add_argument("--selftest", action="store_true") @@ -217,6 +288,17 @@ def main(): a.ebits = 8 if (a.mtp or a.indexer) else 4 if a.xbits is None: a.xbits = a.ebits + # Build per-type bits map. If a type-specific arg is set, use it; otherwise the + # converter falls back to ebits for that type. + bits_map = {} + if a.shared_bits is not None: bits_map["sh"] = a.shared_bits + if a.o_bits is not None: bits_map["o"] = a.o_bits + if a.kvb_bits is not None: bits_map["kvb"] = a.kvb_bits + if a.attn_bits is not None: bits_map["attn"] = a.attn_bits + if a.dmlp_bits is not None: bits_map["dmlp"] = a.dmlp_bits + if bits_map: + print(f"[MIXED] precision map: " + ", ".join(f"{k}={v}bit" for k,v in sorted(bits_map.items()))) + if a.selftest_nvfp4: import torch # 1) LUT e2m1: i 16 codici devono decodificare esattamente ai valori attesi. @@ -298,7 +380,7 @@ def main(): shards = sorted(glob.glob(os.path.join(a.indir, "*.safetensors"))) from safetensors.numpy import save_file for i, sp in enumerate(shards): - out = {}; convert_shard(sp, out, a.n_layers, a.ebits, a.io_bits, a.xbits) + out = {}; convert_shard(sp, out, a.n_layers, a.ebits, a.io_bits, a.xbits, group_size=a.group_size, bits_map=bits_map) save_file(out, os.path.join(a.outdir, f"out-{i:05d}.safetensors")) # copia config + tokenizer for fn in ["config.json"]: @@ -541,7 +623,7 @@ def main(): if os.path.exists(outp): print(f"[MTP] {outp} already done"); continue print(f"[MTP {i+1}/{len(mtp_shards)}] downloading {sh}...", flush=True) p = download_retry(a.repo, sh, tmp) - out = {}; convert_shard(p, out, a.n_layers, a.ebits, a.io_bits, a.xbits, keep_mtp=True) + out = {}; convert_shard(p, out, a.n_layers, a.ebits, a.io_bits, a.xbits, keep_mtp=True, group_size=a.group_size, bits_map=bits_map) save_file(out, outp) os.remove(p) for blob in glob.glob(os.path.join(tmp, "**", "*"), recursive=True): @@ -561,7 +643,7 @@ def main(): if os.path.exists(outp): continue # gia' fatto -> ripartibile print(f"[IDX {i+1}/{len(idx_shards)}] downloading {sh}...", flush=True) p = download_retry(a.repo, sh, tmp) - out = {}; convert_shard(p, out, a.n_layers, a.ebits, a.io_bits, a.xbits, keep_idx=True) + out = {}; convert_shard(p, out, a.n_layers, a.ebits, a.io_bits, a.xbits, keep_idx=True, group_size=a.group_size, bits_map=bits_map) if out: save_file(out, outp) os.remove(p) for blob in glob.glob(os.path.join(tmp, "**", "*"), recursive=True): @@ -575,7 +657,7 @@ def main(): if os.path.exists(outp): continue # gia' fatto -> ripartibile print(f"[{i+1}/{len(shards)}] downloading {sh} ({free_gb(a.outdir):.0f} GB free)...", flush=True) p = download_retry(a.repo, sh, tmp) - out = {}; convert_shard(p, out, a.n_layers, a.ebits, a.io_bits, a.xbits) + out = {}; convert_shard(p, out, a.n_layers, a.ebits, a.io_bits, a.xbits, group_size=a.group_size, bits_map=bits_map) save_file(out, outp) os.remove(p) # <-- cancella subito lo shard fp8 for blob in glob.glob(os.path.join(tmp, "**", "*"), recursive=True): diff --git a/c/tools/eval_glm.py b/c/tools/eval_glm.py index b167253..3aa568e 100644 --- a/c/tools/eval_glm.py +++ b/c/tools/eval_glm.py @@ -48,11 +48,24 @@ def load_docs(task, data_dir, limit, seed): random.Random(seed).shuffle(docs) return docs[:limit] if limit else docs -def build_requests(tk, docs_by_task): +def detect_prefix(snap): + """GLM sees [gMASK] at the start of every training sequence; scoring raw text + without it is out-of-distribution and silently depresses/distorts scores (#108). + Default the prefix ON for GLM snapshots; EVAL_PREFIX (even empty) overrides.""" + if "EVAL_PREFIX" in os.environ: return os.environ["EVAL_PREFIX"] + try: mt = json.load(open(os.path.join(snap, "config.json"))).get("model_type", "") + except Exception: mt = "" + if "glm" in mt.lower(): + print("[prefix] GLM snapshot: prepending [gMASK] to every context " + "(override with EVAL_PREFIX, disable with EVAL_PREFIX=)", file=sys.stderr) + return "[gMASK]" + return "" + +def build_requests(tk, docs_by_task, prefix=""): reqs, meta, perq = [], [], {} for t, docs in docs_by_task.items(): for qi, d in enumerate(docs): - ctx, conts, gold = d["ctx"], d["choices"], int(d["gold"]) + ctx, conts, gold = prefix + d["ctx"], d["choices"], int(d["gold"]) ctx_ids = tk.encode(ctx).ids for oi, cont in enumerate(conts): full = tk.encode(ctx + cont).ids @@ -115,14 +128,17 @@ def main(): docs_by_task = {t: load_docs(t, a.data, a.limit, a.seed) for t in tasks} for t, d in docs_by_task.items(): print(f"[{t}] {len(d)} questions", file=sys.stderr) - reqs, meta, perq = build_requests(tk, docs_by_task) + reqs, meta, perq = build_requests(tk, docs_by_task, detect_prefix(a.snap)) print(f"total requests: {len(reqs)} (answer options)", file=sys.stderr) if a.dry: for r in reqs[:3]: print(" example request:", r[:80], "...", file=sys.stderr) print("DRY: request construction and tokenization passed. Engine was not run.", file=sys.stderr); return - req_path = tempfile.mktemp(suffix=".txt") - open(req_path, "w").write("\n".join(reqs) + "\n") + # mkstemp (non mktemp): crea il file atomicamente con permessi 0600, niente + # race TOCTOU/symlink su una tmp dir condivisa (CWE-377). + fd, req_path = tempfile.mkstemp(suffix=".txt") + with os.fdopen(fd, "w") as f: + f.write("\n".join(reqs) + "\n") env = dict(os.environ, SNAP=a.snap, SCORE=req_path) if a.ram: env["RAM_GB"] = str(a.ram) cmd = [a.glm, str(a.cap)] + a.bits.split() diff --git a/c/tools/expert_atlas/README.md b/c/tools/expert_atlas/README.md new file mode 100644 index 0000000..b1c78ff --- /dev/null +++ b/c/tools/expert_atlas/README.md @@ -0,0 +1,68 @@ +# Expert Atlas — what does each of the 19,456 experts actually do? + +Probe harness for #175. Runs a set of topic-tagged prompts, dumps each run's expert-routing +histogram, and turns them into a per-expert topic-affinity vector. + +```bash +cd c +export COLI_MODEL=/path/to/glm52_i4 +./tools/expert_atlas/sweep.sh # 30 probes (10 topics x 3 prompts) +python3 tools/expert_atlas/analyze.py --stats atlas_out/stats --out atlas_out/experts.json \ + --web web/dist/experts.json # optional: feed the web dashboard Atlas +python3 tools/expert_atlas/validate.py atlas_out/stats 200 # leave-one-prompt-out check +``` + +`--web` writes the same atlas in the shape the web dashboard consumes (the Atlas galaxy and the +Brain hover tooltips): keyed `"layer:expert"` with `affinity`/`entropy`/`top`/`label`. It replaces +the retired `tools/expert_atlas.py`, whose API-driven probing ran through a live server and was +exposed to exactly the traps above (server-side `--topp`, speculative drafts, shared `.coli_usage`). + +## Read this before you trust any atlas + +Four things silently corrupt this measurement. The sweep script controls all of them; if you +roll your own, don't skip them. + +| trap | effect | control | +|---|---|---| +| **`--topp`** | prunes experts by cumulative probability — measured: it hides **38% of the distinct experts** (7,587 → 4,687). It is also the *recommended speed setting*. | `TOPP=0` | +| **speculative drafts** | `eusage` is incremented inside `moe()`, *before* verification, so **rejected** drafts count. Those are experts routed for text the model never emitted. | `MTP=0 DRAFT=0` | +| **`.coli_usage`** | is loaded at startup and accumulates, so a naive `STATS` dump contains **all prior history**, not this run. | remove per run (script backs it up and restores) | +| **autocorrelation** | routing inside one run is highly correlated — the same context routes to the same experts token after token. An expert firing 38 times during one prompt is **one** observation, not 38. Chi-square/entropy on raw selections will certify single-prompt flukes as perfect specialists. | `analyze.py` requires the affinity to **replicate across a category's independent prompts** | + +The CUDA expert tier is also not run-to-run deterministic, so the sweep uses `--gpu none`. Tier +config only decides *where weights live*, not what the router picks, so this costs nothing. + +## Method + +`analyze.py`: + +1. `n[e][c]` — selections of expert *e* while running category *c* +2. `f[e][c] = n[e][c] / N[c]` — normalise by **category size** (prefill routes the prompt too, so a + verbose category would otherwise look busier) +3. `p(c|e)` — renormalise into a topic distribution per expert, i.e. base-rate corrected. Ranking + by raw count instead just rediscovers which experts are popular in general. +4. `spec(e) = 1 − H(p(c|e)) / log C` — 0 = generalist, 1 = fires for exactly one topic +5. **replication gate** — an expert is only a candidate specialist for *c* if it fires in ≥2 of *c*'s + independent prompts + +`validate.py` — leave-one-prompt-out. Build each category's top-K specialist set from its *other* +prompts, then check which set the held-out prompt's routing actually lands in. If specialisation +were an artifact of prompt wording, the held-out prompt would not prefer its own category. + +## Result on GLM-5.2 744B int4 (Zen5, CPU routing path) + +- **Leave-one-prompt-out: 29/30 = 96.7%** (chance 10%). Specialisation is a property of the topic, + not of prompt wording. +- The single miss is instructive: `写一首关于秋天的短诗` ("write a short poem about autumn") is + classified **poetry**, not Chinese — routing follows the **task** over the **language**. +- **Only 7.9% of experts are strong specialists** (spec ≥ 0.5). The "one expert = one topic" + picture is wrong for ~92% of them. +- Specialisation **rises with depth**: layer 3 ≈ 0.07 (generalist, token/syntax level) → layers + 18–58 ≈ 0.19–0.27. +- The replication gate removed **587** experts that looked like flawless specialists on one prompt. + +## Extending it + +`probes.json` is the whole probe set — add categories and prompts. Use **3+ prompts per category +with varied phrasing**, or the replication gate has nothing to check and you are back to measuring +one prompt. Keep prompt lengths in a similar band across categories. diff --git a/c/tools/expert_atlas/analyze.py b/c/tools/expert_atlas/analyze.py new file mode 100644 index 0000000..c5eb502 --- /dev/null +++ b/c/tools/expert_atlas/analyze.py @@ -0,0 +1,140 @@ +#!/usr/bin/env python3 +"""GLM-5.2 Expert Atlas — affinity with CROSS-PROMPT REPLICATION (#175). + +Fixes a trap in the naive version. Routing selections inside one run are heavily correlated: +the same context routes to the same experts token after token. So an expert that fires 38 +times during a single 'code' prompt is ONE effective observation, not 38 independent draws. +Treat them as independent and a chi-square will happily certify a single-prompt fluke as a +perfect specialist (spec=1.000, lift=10.0) on 38 selections. It is measuring one prompt. + +So specialisation is only claimed when it REPLICATES across the independent prompts of a +category: + - each category has R prompts (here 3), each run is one replicate + - share[e][run] = selections of e in that run / total selections in that run + - an expert is a candidate specialist for category c only if it fires in >= MIN_RUNS of + c's runs (default 2/3) -> it is a property of the TOPIC, not of one prompt's wording + - affinity uses the MEAN share across a category's runs, so one hot run cannot carry it + - reliability = (runs in top category) / (runs in that category) + +Also reports the generalist/specialist split by layer depth, which is an average over +thousands of experts and is robust to the above. +""" +import argparse, glob, json, math, os +from collections import defaultdict + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--stats", default="stats") + ap.add_argument("--min-count", type=int, default=30) + ap.add_argument("--min-runs", type=int, default=2, help="must fire in >= this many of the top category's runs") + ap.add_argument("--out", default="experts.json") + ap.add_argument("--web", default="", help="also write the web-dashboard experts.json (Atlas/Brain hover)") + a = ap.parse_args() + + # run[(cat,idx)][(layer,expert)] = count ; run_tot[(cat,idx)] = total + run_counts, run_tot = defaultdict(dict), defaultdict(int) + for path in sorted(glob.glob(os.path.join(a.stats, "*.txt"))): + base = os.path.basename(path)[:-4] + cat, idx = base.rsplit("_", 1) + for line in open(path): + p = line.split() + if len(p) != 3: + continue + l, e, n = int(p[0]), int(p[1]), int(p[2]) + run_counts[(cat, idx)][(l, e)] = n + run_tot[(cat, idx)] += n + + cats = sorted({c for c, _ in run_counts}) + runs_of = {c: sorted(i for cc, i in run_counts if cc == c) for c in cats} + C = len(cats) + print(f"categories ({C}): " + ", ".join(f"{c}[{len(runs_of[c])}]" for c in cats)) + + experts = {k for d in run_counts.values() for k in d} + print(f"experts seen: {len(experts):,}\n") + + atlas, dropped_sparse, dropped_unrepl = [], 0, 0 + for key in experts: + total = sum(run_counts[r].get(key, 0) for r in run_counts) + if total < a.min_count: + dropped_sparse += 1 + continue + # mean share per category across its runs (a single hot run cannot carry the category) + mean_share, fired_runs = {}, {} + for c in cats: + shares, fired = [], 0 + for i in runs_of[c]: + n = run_counts[(c, i)].get(key, 0) + shares.append(n / max(1, run_tot[(c, i)])) + if n > 0: + fired += 1 + mean_share[c] = sum(shares) / len(shares) + fired_runs[c] = fired + s = sum(mean_share.values()) + if s <= 0: + continue + p = {c: mean_share[c] / s for c in cats} + top = max(cats, key=lambda c: p[c]) + # REPLICATION GATE: the affinity must show up in >= min-runs of that category's prompts + if fired_runs[top] < a.min_runs: + dropped_unrepl += 1 + continue + H = -sum(v * math.log(v) for v in p.values() if v > 0) + atlas.append({ + "layer": key[0], "expert": key[1], "total": total, + "spec": round(1.0 - H / math.log(C), 4), + "top_topic": top, + "top_lift": round(p[top] * C, 2), + "reliability": f"{fired_runs[top]}/{len(runs_of[top])}", + "p": {c: round(p[c], 4) for c in cats}, + }) + atlas.sort(key=lambda r: (-r["spec"], -r["total"])) + + print(f"dropped {dropped_sparse:,} sparse (<{a.min_count} sel)") + print(f"dropped {dropped_unrepl:,} UNREPLICATED (fired in <{a.min_runs} runs of their top topic)") + print(f"kept {len(atlas):,} experts\n") + + print("=== most specialised, replicated across prompts ===") + print(f"{'layer':>5} {'exp':>4} {'sel':>6} {'spec':>6} {'lift':>6} {'repl':>5} topic") + for r in atlas[:20]: + print(f"{r['layer']:>5} {r['expert']:>4} {r['total']:>6} {r['spec']:>6.3f} " + f"{r['top_lift']:>6.2f} {r['reliability']:>5} {r['top_topic']}") + + print("\n=== specialisation vs layer depth (mean over experts; robust to the above) ===") + by_layer = defaultdict(list) + for r in atlas: + by_layer[r["layer"]].append(r["spec"]) + ls = sorted(by_layer) + for L in ls[::max(1, len(ls)//13)]: + v = by_layer[L] + print(f" layer {L:>3} n={len(v):>4} spec {sum(v)/len(v):.3f} {'#'*int(60*sum(v)/len(v))}") + + print("\n=== experts owned per topic (replicated only) ===") + own = defaultdict(int) + for r in atlas: + own[r["top_topic"]] += 1 + for c in sorted(own, key=lambda x: -own[x]): + print(f" {c:<14} {own[c]:>5}") + + strong = [r for r in atlas if r["spec"] >= 0.5] + print(f"\nstrong specialists (spec >= 0.5, replicated): {len(strong):,} / {len(atlas):,} " + f"({100*len(strong)/max(1,len(atlas)):.1f}%)") + json.dump({"categories": cats, "experts": atlas}, open(a.out, "w"), indent=1) + print(f"wrote {a.out}") + + if a.web: + # Same atlas, keyed "layer:expert" with per-expert affinity/entropy/top/label — + # the shape the web dashboard consumes (Atlas galaxy, Brain hover). + web = {} + for r in atlas: + aff = {c: v for c, v in r["p"].items() if v > 0} + H = -sum(v * math.log2(v) for v in aff.values()) + web[f"{r['layer']}:{r['expert']}"] = { + "affinity": aff, "entropy": round(H, 2), "top": r["top_topic"], + "label": f"specialist: {r['top_topic']}" if r["spec"] >= 0.5 else "generalist"} + json.dump({"categories": cats, "experts": web}, open(a.web, "w")) + print(f"wrote {a.web} (dashboard format, {len(web):,} experts)") + + +if __name__ == "__main__": + main() diff --git a/c/tools/expert_atlas/probes.json b/c/tools/expert_atlas/probes.json new file mode 100644 index 0000000..ae377da --- /dev/null +++ b/c/tools/expert_atlas/probes.json @@ -0,0 +1,54 @@ +{ + "_comment": "Probe set for the GLM-5.2 Expert Atlas (#175). 10 categories x 3 prompts. Prompts are deliberately varied in phrasing within a category so the affinity vector reflects the TOPIC, not one prompt's surface form. Lengths are kept within a narrow band across categories: prefill routes the prompt tokens too, so a long prompt in one category would inflate its counts.", + + "code_python": [ + "Write a Python function that merges two sorted lists into one sorted list.", + "In Python, explain the difference between a generator and a list comprehension.", + "Refactor this Python snippet to avoid a nested loop: for a in xs: for b in ys: if a==b: out.append(a)" + ], + "code_sql": [ + "Write a SQL query that returns the top 5 customers by total order value.", + "Explain what a LEFT JOIN does differently from an INNER JOIN in SQL.", + "Write SQL to add an index on the email column of a users table and explain when it helps." + ], + "math_proof": [ + "Prove that the square root of 2 is irrational.", + "Show that the sum of the first n odd numbers equals n squared.", + "Explain why the derivative of e^x is e^x." + ], + "chinese": [ + "请用三句话解释什么是机器学习。", + "写一首关于秋天的短诗。", + "中国的四大发明是什么？请简要说明。" + ], + "german": [ + "Erkläre in drei Sätzen, wie ein Verbrennungsmotor funktioniert.", + "Schreibe eine kurze formelle E-Mail, in der du einen Termin absagst.", + "Was ist der Unterschied zwischen Dativ und Akkusativ im Deutschen?" + ], + "poetry": [ + "Write a short poem about the sea at night.", + "Compose four lines of verse about an empty train station.", + "Write a haiku about the first snow of winter." + ], + "law": [ + "Explain in plain terms what consideration means in contract law.", + "What is the difference between a misdemeanor and a felony?", + "Summarize what the term 'force majeure' covers in a commercial contract." + ], + "medicine": [ + "Explain the difference between type 1 and type 2 diabetes.", + "What are the common symptoms of iron deficiency anemia?", + "Describe how a vaccine produces immunity." + ], + "json_format": [ + "Return a JSON object with keys name, age, and email for a fictional user. Output only JSON.", + "Convert this to JSON: name Alice, roles admin and editor, active true. Output only JSON.", + "Write a JSON schema for an object with a required string field 'id' and an optional integer 'count'." + ], + "casual_chat": [ + "Hey, what's a good way to spend a rainy Sunday afternoon?", + "I'm feeling pretty tired today. Any tips to get through the afternoon?", + "What's your favourite kind of weather, and why?" + ] +} diff --git a/c/tools/expert_atlas/sweep.sh b/c/tools/expert_atlas/sweep.sh new file mode 100755 index 0000000..9c101d3 --- /dev/null +++ b/c/tools/expert_atlas/sweep.sh @@ -0,0 +1,82 @@ +#!/usr/bin/env bash +# GLM-5.2 Expert Atlas — probe sweep (#175). +# +# cd c && ./tools/expert_atlas/sweep.sh [probes.json] [outdir] +# +# Env: COLI_MODEL, NGEN (default 64), COLI (default ./coli) +# +# --------------------------------------------------------------------------------------------- +# THE CONFOUNDS THIS SCRIPT EXISTS TO CONTROL. Each one silently corrupts the atlas. +# +# TOPP=0 --topp prunes experts by cumulative probability. Measured on GLM-5.2 int4, +# same prompt, only top-p changed: +# topp=0 -> 21,000 selections across 7,587 distinct experts +# topp=0.7 -> 11,944 selections across 4,687 distinct experts +# It hides 38% of the experts. It is also the RECOMMENDED SPEED SETTING, so this +# is very easy to walk into: with top-p on you profile the pruner, not the model. +# +# MTP=0 Speculative drafts route experts for tokens that are later REJECTED and never +# DRAFT=0 emitted (eusage is incremented inside moe(), before verification). Those counts +# would describe text the model never produced. +# +# --gpu none The CUDA expert tier is not run-to-run deterministic (VRAM placement shifts +# between runs). Routing on the CPU path is reproducible. The tier only decides +# where weights live, not what the router picks, so CPU costs nothing here. +# +# --temp 0 Greedy: deterministic continuation, reproducible atlas. +# +# rm .coli_usage before EVERY run +# eusage is LOADED from /.coli_usage at startup and written back at exit, so +# a naive STATS dump contains ALL PRIOR HISTORY, not this run. Removing it per run +# makes each dump exactly one probe. The user's learned cache is backed up and +# restored on exit. +# +# Prompt lengths are kept in a narrow band across categories: prefill routes the prompt tokens +# too, so a verbose category would otherwise simply look "busier". +# --------------------------------------------------------------------------------------------- +set -uo pipefail +HERE="$(cd "$(dirname "$0")" && pwd)" +PROBES="${1:-$HERE/probes.json}" +OUT="${2:-./atlas_out}" +COLI="${COLI:-./coli}" +NGEN="${NGEN:-64}" +MODEL="${COLI_MODEL:?set COLI_MODEL to the snapshot directory}" + +[ -f "$PROBES" ] || { echo "no probe file: $PROBES" >&2; exit 1; } +[ -x "$COLI" ] || { echo "no coli at $COLI (run from c/, or set COLI=)" >&2; exit 1; } +mkdir -p "$OUT/stats" + +export COLI_MODEL="$MODEL" MTP=0 DRAFT=0 TOPP=0 + +USAGE="$MODEL/.coli_usage" +BACKUP="$OUT/.coli_usage.backup" +[ -f "$USAGE" ] && cp "$USAGE" "$BACKUP" && echo "backed up $USAGE" +restore(){ [ -f "$BACKUP" ] && cp "$BACKUP" "$USAGE" && echo "restored $USAGE"; } +trap restore EXIT + +python3 - "$PROBES" > "$OUT/runlist.tsv" <<'PY' +import json, sys +for cat, prompts in json.load(open(sys.argv[1])).items(): + if cat.startswith('_'): + continue + for i, p in enumerate(prompts): + print(f"{cat}\t{i}\t{p}") +PY + +n=$(wc -l < "$OUT/runlist.tsv"); i=0 +echo "$n probes -> $OUT/stats" +while IFS=$'\t' read -r cat idx prompt; do + i=$((i+1)) + dst="$OUT/stats/${cat}_${idx}.txt" + [ -s "$dst" ] && { echo " [$i/$n] $cat/$idx (cached)"; continue; } + rm -f "$USAGE" # start from an EMPTY routing history + STATS="$dst" "$COLI" run "$prompt" --ngen "$NGEN" --ctx 4096 --gpu none --temp 0 \ + > "$OUT/stats/${cat}_${idx}.log" 2>&1 + echo " [$i/$n] $cat/$idx $(grep -aoE '[0-9]+ selections across [0-9]+ distinct experts' \ + "$OUT/stats/${cat}_${idx}.log" | head -1)" +done < "$OUT/runlist.tsv" + +echo +echo "next:" +echo " python3 $HERE/analyze.py --stats $OUT/stats --out $OUT/experts.json" +echo " python3 $HERE/validate.py $OUT/stats 200" diff --git a/c/tools/expert_atlas/validate.py b/c/tools/expert_atlas/validate.py new file mode 100644 index 0000000..6b283da --- /dev/null +++ b/c/tools/expert_atlas/validate.py @@ -0,0 +1,79 @@ +#!/usr/bin/env python3 +"""Leave-one-prompt-out validation of the Expert Atlas (#175). + +"Replicates across the 3 prompts I picked" is not the same as "generalises". The atlas is +only real if a specialist set learned from SOME prompts predicts routing on a prompt it has +never seen. + +Protocol, for every category c and every held-out prompt h of c: + 1. build c's top-K specialist set from c's OTHER prompts only (h excluded entirely) + 2. do the same for all other categories (using ALL their prompts — they never saw h either) + 3. on the held-out run h, measure what share of routing selections land in each set + 4. the atlas works if c's own set wins on h + +If specialisation were an artifact of prompt wording, the held-out prompt would not prefer +its own category's set. Chance is 1/C. +""" +import glob, json, os, sys +from collections import defaultdict + +STATS = sys.argv[1] if len(sys.argv) > 1 else "stats" +K = int(sys.argv[2]) if len(sys.argv) > 2 else 200 # specialists per category + +runs, tot = {}, {} +for path in sorted(glob.glob(os.path.join(STATS, "*.txt"))): + cat, idx = os.path.basename(path)[:-4].rsplit("_", 1) + d = {} + t = 0 + for line in open(path): + p = line.split() + if len(p) == 3: + d[(int(p[0]), int(p[1]))] = int(p[2]) + t += int(p[2]) + runs[(cat, idx)] = d + tot[(cat, idx)] = t + +cats = sorted({c for c, _ in runs}) +idxs = {c: sorted(i for cc, i in runs if cc == c) for c in cats} +C = len(cats) + + +def specialists(cat, exclude): + """Top-K experts by lift for `cat`, computed WITHOUT the excluded run.""" + share = defaultdict(lambda: defaultdict(float)) + for c in cats: + used = [i for i in idxs[c] if not (c == cat and i == exclude)] + for i in used: + for k, n in runs[(c, i)].items(): + share[k][c] += n / max(1, tot[(c, i)]) / len(used) + scored = [] + for k, per in share.items(): + s = sum(per.values()) + if s <= 0: + continue + p = per[cat] / s + if per[cat] > 0: + scored.append((p, k)) + scored.sort(reverse=True) + return {k for _, k in scored[:K]} + + +print(f"leave-one-prompt-out, {C} categories, top-{K} specialists per category") +print(f"chance = {100.0/C:.1f}%\n") +hits = 0 +trials = 0 +for c in cats: + for h in idxs[c]: + sets = {cc: specialists(cc, h if cc == c else None) for cc in cats} + held = runs[(c, h)] + htot = max(1, tot[(c, h)]) + scores = {cc: sum(held.get(k, 0) for k in sets[cc]) / htot for cc in cats} + win = max(scores, key=scores.get) + ok = win == c + hits += ok + trials += 1 + own = 100 * scores[c] + best_other = 100 * max(v for cc, v in scores.items() if cc != c) + print(f" {c:<12} prompt {h} own-set {own:5.2f}% best-other {best_other:5.2f}% " + f"-> {'HIT ' if ok else 'MISS'} (predicted {win})") +print(f"\naccuracy: {hits}/{trials} = {100*hits/trials:.1f}% (chance {100.0/C:.1f}%)") diff --git a/c/tools/fetch_benchmarks.py b/c/tools/fetch_benchmarks.py index 6d2abd2..05dd9e7 100644 --- a/c/tools/fetch_benchmarks.py +++ b/c/tools/fetch_benchmarks.py @@ -8,7 +8,7 @@ USO: Poi: python3 tools/eval_glm.py --snap /home/vincenzo/glm52_i4 --data ./bench --tasks mmlu --limit 40 --ram 15 """ -import os, json, argparse, random +import os, sys, json, time, argparse, random def f_hellaswag(d): ctx = (d["activity_label"] + ": " + d["ctx_a"] + " " + d["ctx_b"].capitalize()).strip() @@ -38,19 +38,44 @@ TASKS = { # task: (path, config, split, formatter) "openbookqa": ("allenai/openbookqa", "main", "validation", f_openbookqa), } +def load_retry(path, cfg, split, tries=5): + """L'hub restituisce 5xx/timeout transitori (#304): riprova con backoff invece di + morire al primo HEAD fallito. hf_hub riprende i download parziali dalla cache, + quindi il retry riparte da dove si era fermato, non da zero. + EN: the hub throws transient 5xx/timeouts (#304): retry with backoff instead of + dying on the first failed HEAD. hf_hub resumes partial downloads from its cache, + so a retry continues where it stopped rather than starting over.""" + from datasets import load_dataset + for k in range(tries): + try: + return load_dataset(path, cfg, split=split) + except (KeyboardInterrupt, SystemExit): + raise + except Exception as e: + if k == tries - 1: raise + wait = 2 ** (k + 1) + print(f" {path}: {type(e).__name__}: {e} — retry {k+1}/{tries-1} in {wait}s") + time.sleep(wait) + def main(): ap = argparse.ArgumentParser() ap.add_argument("--out", default="./bench") ap.add_argument("--tasks", default="hellaswag,arc_challenge,mmlu") ap.add_argument("--limit", type=int, default=300) ap.add_argument("--seed", type=int, default=1234) + ap.add_argument("--tries", type=int, default=5) a = ap.parse_args() - from datasets import load_dataset os.makedirs(a.out, exist_ok=True) + failed = [] for t in [x.strip() for x in a.tasks.split(",") if x.strip()]: - if t not in TASKS: print("unknown task:", t); continue + if t not in TASKS: print("unknown task:", t); failed.append(t); continue path, cfg, split, fn = TASKS[t] - ds = load_dataset(path, cfg, split=split) + try: + ds = load_retry(path, cfg, split, tries=max(a.tries, 1)) + except Exception as e: + # un task fallito non deve uccidere gli altri / one failed task must not kill the rest + print(f"{t}: FAILED after {a.tries} tries ({type(e).__name__}: {e}) — skipping") + failed.append(t); continue idx = list(range(len(ds))); random.Random(a.seed).shuffle(idx) rows, n = [], 0 for i in idx: @@ -61,9 +86,18 @@ def main(): except Exception: continue if n >= a.limit: break outp = os.path.join(a.out, t + ".jsonl") - with open(outp, "w") as f: + # scrittura atomica: coli controlla solo l'ESISTENZA del file, quindi un jsonl + # troncato da un run interrotto bloccherebbe il re-download per sempre. + # EN: atomic write: coli only checks the file EXISTS, so a truncated jsonl from + # an interrupted run would block re-download forever. + tmp = outp + ".part" + with open(tmp, "w") as f: for r in rows: f.write(json.dumps(r) + "\n") + os.replace(tmp, outp) print(f"{t}: {len(rows)} -> {outp}") + if failed: + print(f"incomplete: {', '.join(failed)} — rerun when the hub recovers (cached progress is kept)") + sys.exit(1) if __name__ == "__main__": main() diff --git a/c/tools/glm_fp8_emit.py b/c/tools/glm_fp8_emit.py new file mode 100644 index 0000000..45bb304 --- /dev/null +++ b/c/tools/glm_fp8_emit.py @@ -0,0 +1,137 @@ +"""Helper: salva pesi in FP8 e4m3 + scale a blocchi 128x128, nello STESSO layout del +checkpoint reale GLM-5.2-FP8 che `convert_fp8_to_int4.py` legge. + +Layout (deve combaciare col `dequant()` del converter, convert_fp8_to_int4.py:164-169): + - `name` F8_E4M3 [O, I] + - `name_scale_inv` F32 [ceil(O/128), ceil(I/128)] (NOTA: '_scale_inv', underscore) + dequant: W = q.float() * scale.repeat_interleave(128,0).repeat_interleave(128,1)[:O,:I] + +Convenzione FBGEMM/TransformerEngine: scale = amax(blocco)/448 (448 = max e4m3), +si MEMORIZZA il valore e si MOLTIPLICA in dequant. Malgrado il nome "_scale_inv" il +checkpoint memorizza la scala (non il reciproco): e' un MOLTIPLIER. + +EN: Helper that writes weights as FP8 e4m3 with 128x128 block scales, in the SAME layout +EN: as the real GLM-5.2-FP8 checkpoint that `convert_fp8_to_int4.py` reads. +EN: FBGEMM/TransformerEngine convention: scale = amax(block)/448, stored (not its +EN: reciprocal) and MULTIPLIED on dequant. Despite the name "_scale_inv" it is a multiplier. +""" +import torch + +E4M3_MAX = 448.0 # max valore rappresentabile in float8_e4m3fn / max representable value +BLOCK = 128 # granularita' delle scale a blocchi del checkpoint FP8 / FP8 block scale granularity + + +def keep_f32(name, t): + """Stesso set F32 di `classify()` in convert_fp8_to_int4.py (norme, router, bias 1-D). + Tutti gli altri tensori 2-D vengono quantizzati FP8 (attn/mlp/shared/expert/embed/lm_head). + EN: Same F32 set as the converter's classify(): norms, router, 1-D biases. All other 2-D + EN: tensors are FP8-quantized (attn/mlp/shared/expert/embed/lm_head).""" + if t.dim() < 2: + return True # bias 1-D, e_score_correction_bias + if name.endswith("e_score_correction_bias"): + return True + if name.endswith("mlp.gate.weight"): + return True # router (NON gate_proj): tenuto F32 / kept F32 + if name.endswith("norm.weight") or name == "model.norm.weight": + return True # RMSNorm + return False + + +def fp8_block_quantize(w): + """w: [O,I] f32 -> (w_fp8 float8_e4m3fn [O,I], scale_inv f32 [ceil(O/128),ceil(I/128)]). + Identica matematica al `--selftest` del converter (scale = amax(blocco)/448). Padda a + multipli di 128 internamente (gli zeri non alzano l'amax) e fa slice al risultato. + EN: same math as the converter's --selftest. Pads to 128 multiples internally (zeros do + EN: not raise amax), slices the result back to [O,I].""" + O, I = w.shape + nbO, nbI = (O + BLOCK - 1) // BLOCK, (I + BLOCK - 1) // BLOCK + Op, Ip = nbO * BLOCK, nbI * BLOCK + wpad = torch.zeros(Op, Ip, dtype=torch.float32, device=w.device) + wpad[:O, :I] = w + wb = wpad.view(nbO, BLOCK, nbI, BLOCK) # [nbO, BLOCK, nbI, BLOCK] + amax = wb.abs().amax(dim=(1, 3)) # [nbO, nbI] + scale = amax / E4M3_MAX # FBGEMM/TE: memorizza la scala / store the scale + scale = torch.where(scale == 0, torch.ones_like(scale), scale) # blocco tutto-zero -> no div0 + scale = scale.to(torch.float32) + q = (wpad / scale.repeat_interleave(BLOCK, 0).repeat_interleave(BLOCK, 1)).clamp(-E4M3_MAX, E4M3_MAX) + w_fp8 = q.to(torch.float8_e4m3fn) + return w_fp8[:O, :I].contiguous(), scale.contiguous() + + +def fp8_block_dequantize(w_fp8, scale): + """Esatto inverso di fp8_block_quantize, e identico al `dequant()` del converter. + EN: exact inverse of fp8_block_quantize, identical to the converter's dequant().""" + O, I = w_fp8.shape + qf = w_fp8.to(torch.float32) + return qf * scale.repeat_interleave(BLOCK, 0).repeat_interleave(BLOCK, 1)[:O, :I] + + +def unfuse_experts(sd): + """Split HF's fused 3-D `experts.gate_up_proj` [E, 2*M, I] into per-expert 2-D + `experts.{e}.gate_proj` [M, I] + `experts.{e}.up_proj` [M, I], and + `experts.down_proj` [E, I, M] -> `experts.{e}.down_proj` [M_out, I]. + + The real GLM-5.2-FP8 checkpoint stores experts UNFUSED as per-expert 2-D tensors + (gate_proj, up_proj, down_proj), each with its own _scale_inv. HF's + GlmMoeDsaForCausalLM fuses gate+up into a single 3-D gate_up_proj for efficiency. + The converter (classify + ndim!=2 guard) and the C engine both expect the unfused + layout, so we split before saving. + + Idempotent: if experts are already unfused (no 3-D gate_up_proj), returns sd as-is. + EN: split HF's fused 3-D expert weights into the per-expert 2-D layout that the real + EN: checkpoint uses and the converter/engine expect. No-op if already unfused.""" + keys_to_remove = [] + new_entries = {} + for name, t in sd.items(): + if not name.endswith(".mlp.experts.gate_up_proj"): + continue + # prefix = everything before ".mlp.experts.gate_up_proj" + prefix = name[:-len(".mlp.experts.gate_up_proj")] + E, twoM, I = t.shape # [E, 2*intermediate, input] + M = twoM // 2 + for e in range(E): + new_entries[f"{prefix}.mlp.experts.{e}.gate_proj.weight"] = t[e, :M, :].contiguous() + new_entries[f"{prefix}.mlp.experts.{e}.up_proj.weight"] = t[e, M:, :].contiguous() + keys_to_remove.append(name) + # down_proj may be 3-D [E, I, M] in the fused form, or already per-expert + for name, t in sd.items(): + if not name.endswith(".mlp.experts.down_proj") or t.dim() != 3: + continue + prefix = name[:-len(".mlp.experts.down_proj")] + E = t.shape[0] + for e in range(E): + new_entries[f"{prefix}.mlp.experts.{e}.down_proj.weight"] = t[e].contiguous() + keys_to_remove.append(name) + for k in keys_to_remove: + sd.pop(k, None) + sd.update(new_entries) + return sd + + +def state_dict_to_fp8(sd): + """Converte uno state_dict HuggingFace nel layout FP8 del checkpoint reale: + per ogni tensore quantizzabile 2-D scrive `{name}` (F8_E4M3) + `{name}_scale_inv` (F32); + norme/router/bias e qualsiasi tensore NON 2-D (es. pesi MLA impaccati 3-D) restano nel + dtype originale. Questo rispecchia il guard `w.ndim != 2 -> f32` del converter + (convert_fp8_to_int4.py:184). EN: builds the real-checkpoint FP8 layout. Only exactly-2-D + tensors are FP8-quantized; anything else (1-D, 3-D packed MLA weights, ...) is kept, exactly + like the converter's `ndim != 2 -> f32` guard.""" + out = {} + for name, t in sd.items(): + if keep_f32(name, t) or t.dim() != 2: + out[name] = t # f32 / 1-D / 3-D+: tieni / keep + else: + w_fp8, scale = fp8_block_quantize(t.float()) + out[name] = w_fp8 + out[name + "_scale_inv"] = scale + return out + + +def save_fp8_safetensors(sd, path): + """Quantizza a blocchi FP8 e salva in un singolo safetensors leggibile dal converter + via `--indir`. EN: block-quantize to FP8 and save a single safetensors for the converter.""" + from safetensors.torch import save_file + out = state_dict_to_fp8(sd) + save_file({k: v.contiguous() for k, v in out.items()}, str(path)) + n_fp8 = sum(1 for v in out.values() if v.dtype == torch.float8_e4m3fn) + return n_fp8, len(out) diff --git a/c/tools/make_glm_bench_model.py b/c/tools/make_glm_bench_model.py index 3aa1ce9..5ec3457 100644 --- a/c/tools/make_glm_bench_model.py +++ b/c/tools/make_glm_bench_model.py @@ -3,15 +3,27 @@ This is not a useful language model. It preserves the real glm_moe_dsa data flow while remaining small enough to generate locally and run repeated CPU/CUDA A/B tests without downloading the 379 GB checkpoint. + +With --fp8 the weights are written as FP8 e4m3 + 128x128 block scale_inv, in the +SAME layout as the real GLM-5.2-FP8 checkpoint, so convert_fp8_to_int4.py can +exercise its FP8->int4 dequant path on a local fixture (its dims are 128-friendly, +so this is also the right fixture for --group-size 128 testing): + + python tools/make_glm_bench_model.py --fp8 --output glm_bench_fp8 + python tools/convert_fp8_to_int4.py --indir glm_bench_fp8 --outdir glm_bench_i4 --ebits 4 --group-size 128 """ import argparse import json +import sys from pathlib import Path import torch from transformers import GlmMoeDsaConfig, GlmMoeDsaForCausalLM +sys.path.insert(0, str(Path(__file__).resolve().parent)) # importa glm_fp8_emit se lanciato da c/ +from glm_fp8_emit import save_fp8_safetensors, unfuse_experts + def build_config() -> GlmMoeDsaConfig: return GlmMoeDsaConfig( @@ -51,6 +63,9 @@ def main() -> None: parser.add_argument("--output", default="glm_bench_medium") parser.add_argument("--device", default="cuda" if torch.cuda.is_available() else "cpu") parser.add_argument("--seed", type=int, default=1234) + parser.add_argument("--fp8", action="store_true", + help="write weights as FP8 e4m3 + 128x128 block scale_inv (same layout as " + "GLM-5.2-FP8) instead of bf16, so convert_fp8_to_int4.py can dequant+requant") args = parser.parse_args() torch.manual_seed(args.seed) @@ -70,7 +85,6 @@ def main() -> None: output = Path(args.output) output.mkdir(parents=True, exist_ok=True) params = sum(p.numel() for p in model.parameters()) - model.save_pretrained(output, safe_serialization=True, max_shard_size="4GB") model.to(args.device) prompt = [3, 14, 159, 26, 53, 58, 200, 11, 77, 240, 5, 99] @@ -79,6 +93,25 @@ def main() -> None: full = model.generate(ids, max_new_tokens=8, do_sample=False, use_cache=True)[0] logits = model(full.unsqueeze(0), use_cache=False).logits[0] + # Unfuse experts AFTER reference generation (model needs fused weights for + # forward/generate) but BEFORE saving — the real checkpoint and the converter + # + C engine all expect per-expert 2-D gate_proj/up_proj/down_proj tensors. + sd = model.state_dict() + unfuse_experts(sd) + + if args.fp8: + n_fp8, n_tot = save_fp8_safetensors(sd, output / "model.safetensors") + # save_pretrained scrive config.json; nel path FP8 lo bypassiamo, quindi lo scriviamo + # a mano (serve al converter e al motore C). EN: save_pretrained writes config.json; + # the FP8 path bypasses it, so write it manually (converter + C engine need it). + (output / "config.json").write_text(json.dumps(cfg.to_dict())) + print(f"saved FP8: {n_fp8} e4m3 tensors (+{n_tot - n_fp8} scale_inv sidecars / f32) " + f"-> {output / 'model.safetensors'}") + else: + from safetensors.torch import save_file + save_file({k: v.contiguous() for k, v in sd.items()}, str(output / "model.safetensors")) + (output / "config.json").write_text(json.dumps(cfg.to_dict())) + ref = { "prompt_ids": prompt, "full_ids": full.cpu().tolist(), @@ -89,6 +122,7 @@ def main() -> None: "seed": args.seed, "parameters": params, "parameters_billions": round(params / 1e9, 4), + "format": "fp8-e4m3-128" if args.fp8 else "bf16", "purpose": "backend benchmark fixture; random weights, not a language model", } (output / "bench_manifest.json").write_text(json.dumps(manifest, indent=2)) diff --git a/c/tools/make_glm_oracle.py b/c/tools/make_glm_oracle.py index 65a1c19..b623faf 100644 --- a/c/tools/make_glm_oracle.py +++ b/c/tools/make_glm_oracle.py @@ -3,10 +3,34 @@ Architettura vera (MLA + DSA indexer + router sigmoid/noaux_tc + shared expert), dimensioni minuscole. Salva pesi+config in c/glm_tiny/ e un riferimento greedy in c/ref_glm.json. seq corta (<= index_topk) cosi' il DSA seleziona tutte le key e l'attenzione coincide con la MLA densa: il motore C puo' validare senza implementare -l'indexer sparso.""" -import json, torch +l'indexer sparso. + +--fp8: salva i pesi come FP8 e4m3 + scale a blocchi 128x128 (layout del checkpoint reale +GLM-5.2-FP8) invece di bf16, cosi' convert_fp8_to_int4.py puo' esercitare il path FP8->int4 +su un modello minuscolo. PRIMA di calcolare ref_glm.json fa il round-trip dei pesi per FP8 +(quant->dequant, copy_ nel modello): cosi' il riferimento riflette ESATTAMENTE il modello +FP8 che il converter legge, non il modello bf16 a precisione piena. Default: bf16 (oracolo +originale invariato). +EN: --fp8 writes FP8 e4m3 + 128x128 block scale_inv (real GLM-5.2-FP8 layout) instead of bf16, +EN: so convert_fp8_to_int4.py can run its FP8->int4 path on a tiny model. ref_glm.json is +EN: computed AFTER the FP8 round-trip, so the reference matches exactly what the converter +EN: ingests. Default: bf16 (original oracle unchanged).""" +import json, sys, argparse +from pathlib import Path +import torch from transformers import GlmMoeDsaConfig, GlmMoeDsaForCausalLM +sys.path.insert(0, str(Path(__file__).resolve().parent)) # importa glm_fp8_emit se lanciato da c/ +from glm_fp8_emit import (fp8_block_quantize, fp8_block_dequantize, keep_f32, + save_fp8_safetensors, unfuse_experts) + +ap = argparse.ArgumentParser() +ap.add_argument("--fp8", action="store_true", + help="salva in FP8 e4m3 + 128x128 block scale_inv (layout GLM-5.2-FP8) e " + "calcola ref_glm.json sul modello dopo il round-trip FP8. " + "EN: write FP8 e4m3 + block scale_inv, ref computed on FP8-rounded model") +args = ap.parse_args() + torch.manual_seed(1234) cfg = GlmMoeDsaConfig( @@ -53,6 +77,18 @@ with torch.no_grad(): layer.mlp.gate.e_score_correction_bias.copy_( torch.linspace(-0.1, 0.1, cfg.n_routed_experts)) +# --fp8: round-trip dei pesi quantizzabili per FP8 PRIMA di calcolare il riferimento, +# cosi' ref_glm.json riflette esattamente il modello FP8 che il converter leggera'. +# Norme/router/bias (keep_f32) restano a precisione piena. EN: --fp8: round-trip quantizable +# weights through FP8 before computing the reference, so ref_glm.json matches the FP8 model. +if args.fp8: + with torch.no_grad(): + for n, p in model.named_parameters(): + if keep_f32(n, p) or p.dim() != 2: + continue + q, s = fp8_block_quantize(p) + p.copy_(fp8_block_dequantize(q, s)) + print("=== state_dict tensors (names used by the C loader) ===") for n, p in model.state_dict().items(): print(f" {n:60s} {tuple(p.shape)}") @@ -73,7 +109,20 @@ with torch.no_grad(): tf_pred = lg.argmax(-1).tolist() print("tf_pred:", tf_pred) -model.save_pretrained("glm_tiny", safe_serialization=True) +# Unfuse experts AFTER reference generation (model needs fused weights for +# forward/generate) but BEFORE saving — the real checkpoint and the converter +# + C engine all expect per-expert 2-D gate_proj/up_proj/down_proj tensors. +sd = model.state_dict() +unfuse_experts(sd) + +if args.fp8: + n_fp8, n_tot = save_fp8_safetensors(sd, "glm_tiny/model.safetensors") + print(f"\nsaved FP8: {n_fp8} e4m3 tensors (+{n_tot - n_fp8} scale_inv sidecars / f32) " + f"-> glm_tiny/model.safetensors") +else: + from safetensors.torch import save_file + save_file({k: v.contiguous() for k, v in sd.items()}, "glm_tiny/model.safetensors") json.dump(cfg.to_dict(), open("glm_tiny/config.json", "w")) json.dump({"prompt_ids": prompt, "full_ids": full, "tf_pred": tf_pred}, open("ref_glm.json", "w")) -print("\nsaved: glm_tiny/ (weights + config) and ref_glm.json") +print("saved: glm_tiny/ (weights + config) and ref_glm.json" + + (" [fp8]" if args.fp8 else "")) diff --git a/c/tools/quant_ablation.py b/c/tools/quant_ablation.py index 7a870ff..689643a 100644 --- a/c/tools/quant_ablation.py +++ b/c/tools/quant_ablation.py @@ -106,33 +106,106 @@ def rotation(dim, device, seed=417): return q -def quantize_param(w, bits, group, rot=False): +def quantize_param(w, bits, group, rot=False, e8=""): if w.ndim == 3: # fused experts [E, in, out] -> move input last x = w.transpose(1, 2).contiguous() - x = _rot_quant(x, bits, group) if rot else _quant_last_dim(x, bits, group) + x = _rot_quant(x, bits, group, e8) if rot else _grid_or_e8(x, bits, group, e8) return x.transpose(1, 2).contiguous() if rot: - return _rot_quant(w, bits, group) - return _quant_last_dim(w, bits, group) # nn.Linear [out, in] -- input already last + return _rot_quant(w, bits, group, e8) + return _grid_or_e8(w, bits, group, e8) # nn.Linear [out, in] -- input already last -def _rot_quant(x, bits, group): +def _grid_or_e8(x, bits, group, e8): + if e8: + return _quant_e8(x.float(), group, ball=(e8 == "-e8")) + return _quant_last_dim(x, bits, group) + + +def _rot_quant(x, bits, group, e8=""): """W -> Qn(W@Q) @ Q^T along the last (input) dim — see rotation() above.""" q = rotation(x.shape[-1], x.device) - return (_quant_last_dim(x.float() @ q, bits, group) @ q.T).contiguous() + return (_grid_or_e8(x.float() @ q, bits, group, e8) @ q.T).contiguous() -SCHEME_RE = re.compile(r"^int(2|3|4|8)(?:-g(\d+))?(-rot)?(-nohead)?$") +# -------------------------------------------------------------------------------------- +# E8 lattice quantization (#81 follow-up): the -rot schemes above are QuaRot (rotation + +# uniform grid). QuIP#'s 2-bit result needs the second ingredient — an E8 lattice codebook +# instead of the grid. E8 = D8 ∪ (D8 + 1/2), nearest point via Conway–Sloane: round every +# coordinate, and if the sum is odd re-round the worst coordinate the other way; repeat on +# the half-shifted copy and keep the closer of the two. `-e8` clamps points to |p|^2 <= 10, +# the E8P ball QuIP# builds its 2^16 codebook from (2 bits/weight for 8-dim blocks); +# `-e8u` leaves the lattice unbounded — an ideal-codebook upper bound, not a deployable rate. +# Scale: per group, a small MSE search over multiples of the block RMS (absmax is the wrong +# statistic for a lattice — the ball wants energy matched, not the peak). +# -------------------------------------------------------------------------------------- +def _d8_nearest(y): + f = torch.round(y) + d = y - f + odd = (f.sum(-1).long() & 1).bool() + idx = d.abs().argmax(-1, keepdim=True) + step = torch.where(d.gather(-1, idx) >= 0, 1.0, -1.0) + flipped = f.gather(-1, idx) + step + return f.scatter(-1, idx, torch.where(odd[..., None], flipped, f.gather(-1, idx))) + + +def _e8_nearest(y): + a = _d8_nearest(y) + b = _d8_nearest(y - 0.5) + 0.5 + da = ((y - a) ** 2).sum(-1, keepdim=True) + db = ((y - b) ** 2).sum(-1, keepdim=True) + return torch.where(da <= db, a, b) + + +def _e8_ball(y, r2=10.0): + p = _e8_nearest(y) + for _ in range(8): # shrink-and-requantize until inside + n2 = (p ** 2).sum(-1, keepdim=True) + over = n2 > r2 + 1e-6 + if not over.any(): + break + y = torch.where(over, y * torch.sqrt(r2 / torch.clamp(n2, min=r2)) * 0.98, y) + p = torch.where(over, _e8_nearest(y), p) + return p + + +def _quant_e8(x, group, ball): + """Blocks of 8 along the input dim; per-group scale by MSE search over RMS multiples.""" + if x.shape[-1] % 8: + raise SystemExit(f"-e8 needs input dim divisible by 8 (got {x.shape[-1]})") + g = group or x.shape[-1] + if g % 8: + raise SystemExit(f"-e8 group {g} must be a multiple of 8") + shp = x.shape + xg = x.reshape(-1, g) # [G, g] + rms = torch.clamp(xg.pow(2).mean(-1, keepdim=True).sqrt(), min=1e-8) + best_out, best_err = None, None + for k in (0.5, 0.7, 0.9, 1.1, 1.4, 1.8, 2.4): + s = rms * k + yb = (xg / s).reshape(-1, g // 8, 8) + p = _e8_ball(yb) if ball else _e8_nearest(yb) + out = (p.reshape(-1, g) * s) + err = (out - xg).pow(2).sum(-1, keepdim=True) + if best_err is None: + best_out, best_err = out, err + else: + take = err < best_err + best_out = torch.where(take, out, best_out) + best_err = torch.where(take, err, best_err) + return best_out.reshape(shp) + + +SCHEME_RE = re.compile(r"^int(2|3|4|8)(?:-g(\d+))?(-e8u?)?(-rot)?(-nohead)?$") def parse_scheme(name): - """'int4-g128-nohead' -> (bits=4, group=128, skip_head=True). 'fp16' -> None.""" + """'int4-g128-nohead' -> (bits, group, e8, skip_head...). 'fp16' -> None.""" if name == "fp16": return None m = SCHEME_RE.match(name) if not m: - raise SystemExit(f"bad scheme '{name}' (expected fp16 | int{{2,3,4,8}}[-g][-rot][-nohead])") - return int(m.group(1)), int(m.group(2) or 0), bool(m.group(3)), bool(m.group(4)) + raise SystemExit(f"bad scheme '{name}' (expected fp16 | int{{2,3,4,8}}[-g][-e8|-e8u][-rot][-nohead])") + return int(m.group(1)), int(m.group(2) or 0), m.group(3) or "", bool(m.group(4)), bool(m.group(5)) def is_router(name): @@ -152,7 +225,7 @@ def apply_scheme(model, scheme): spec = parse_scheme(scheme) if spec is None: return 0, 0, total - bits, group, rot, skip_head = spec + bits, group, e8, rot, skip_head = spec n = qp = 0 with torch.no_grad(): for name, p in model.named_parameters(): @@ -160,7 +233,7 @@ def apply_scheme(model, scheme): continue if skip_head and is_head_or_embed(name): continue - p.data.copy_(quantize_param(p.data.float(), bits, group, rot).to(p.dtype)) + p.data.copy_(quantize_param(p.data.float(), bits, group, rot, e8).to(p.dtype)) n += 1 qp += p.numel() return n, qp, total diff --git a/c/tools/run_tests.py b/c/tools/run_tests.py index e2c17ac..1aee5a4 100644 --- a/c/tools/run_tests.py +++ b/c/tools/run_tests.py @@ -5,12 +5,13 @@ Used by `make test-c` so the test runner works from any shell (cmd.exe, PowerShell, Git Bash, MSYS2) without a POSIX `for` loop. Each test binary is a positional argument. """ +import os import subprocess import sys failed = [] for binary in sys.argv[1:]: - rc = subprocess.call([binary]) + rc = subprocess.call([os.path.normpath(binary)]) if rc != 0: failed.append(binary) if failed: diff --git a/c/uring.h b/c/uring.h new file mode 100644 index 0000000..6235234 --- /dev/null +++ b/c/uring.h @@ -0,0 +1,137 @@ +#ifndef COLI_URING_H +#define COLI_URING_H + +/* Minimal Linux io_uring reader. Colibri owns the ring from one thread, queues + * a batch of positioned reads, and reaps CQEs without a userspace spin loop. */ +#ifdef __linux__ + +#include +#include +#include +#include +#include +#include +#include + +#ifndef MAP_POPULATE +#define MAP_POPULATE 0 +#endif + +typedef struct { + int fd; + struct io_uring_params p; + void *sq_map, *cq_map, *sqes_map; + size_t sq_map_sz, cq_map_sz, sqes_map_sz; + unsigned *sq_head, *sq_tail, *sq_mask, *sq_entries, *sq_array; + unsigned *cq_head, *cq_tail, *cq_mask, *cq_entries; + struct io_uring_sqe *sqes; + struct io_uring_cqe *cqes; +} ColiUring; + +static inline unsigned coli_uring_load_acquire(unsigned *p){ + return __atomic_load_n(p,__ATOMIC_ACQUIRE); +} +static inline void coli_uring_store_release(unsigned *p,unsigned v){ + __atomic_store_n(p,v,__ATOMIC_RELEASE); +} + +static inline void coli_uring_close(ColiUring *r){ + if(!r) return; + if(r->sqes_map && r->sqes_map!=MAP_FAILED) munmap(r->sqes_map,r->sqes_map_sz); + if(r->cq_map && r->cq_map!=MAP_FAILED && r->cq_map!=r->sq_map) munmap(r->cq_map,r->cq_map_sz); + if(r->sq_map && r->sq_map!=MAP_FAILED) munmap(r->sq_map,r->sq_map_sz); + if(r->fd>=0) close(r->fd); + memset(r,0,sizeof(*r)); r->fd=-1; +} + +static inline int coli_uring_init(ColiUring *r,unsigned entries){ + memset(r,0,sizeof(*r)); r->fd=-1; + r->fd=(int)syscall(SYS_io_uring_setup,entries,&r->p); + if(r->fd<0) return -1; + + r->sq_map_sz=r->p.sq_off.array+r->p.sq_entries*sizeof(unsigned); + r->cq_map_sz=r->p.cq_off.cqes+r->p.cq_entries*sizeof(struct io_uring_cqe); + if(r->p.features&IORING_FEAT_SINGLE_MMAP){ + size_t n=r->sq_map_sz>r->cq_map_sz?r->sq_map_sz:r->cq_map_sz; + r->sq_map=mmap(NULL,n,PROT_READ|PROT_WRITE,MAP_SHARED|MAP_POPULATE,r->fd,IORING_OFF_SQ_RING); + if(r->sq_map==MAP_FAILED){ r->sq_map=NULL; coli_uring_close(r); return -1; } + r->sq_map_sz=n; r->cq_map=r->sq_map; r->cq_map_sz=n; + }else{ + r->sq_map=mmap(NULL,r->sq_map_sz,PROT_READ|PROT_WRITE,MAP_SHARED|MAP_POPULATE,r->fd,IORING_OFF_SQ_RING); + if(r->sq_map==MAP_FAILED){ r->sq_map=NULL; coli_uring_close(r); return -1; } + r->cq_map=mmap(NULL,r->cq_map_sz,PROT_READ|PROT_WRITE,MAP_SHARED|MAP_POPULATE,r->fd,IORING_OFF_CQ_RING); + if(r->cq_map==MAP_FAILED){ r->cq_map=NULL; coli_uring_close(r); return -1; } + } + r->sqes_map_sz=r->p.sq_entries*sizeof(struct io_uring_sqe); + r->sqes_map=mmap(NULL,r->sqes_map_sz,PROT_READ|PROT_WRITE,MAP_SHARED|MAP_POPULATE,r->fd,IORING_OFF_SQES); + if(r->sqes_map==MAP_FAILED){ r->sqes_map=NULL; coli_uring_close(r); return -1; } + + char *sq=(char*)r->sq_map,*cq=(char*)r->cq_map; + r->sq_head=(unsigned*)(sq+r->p.sq_off.head); + r->sq_tail=(unsigned*)(sq+r->p.sq_off.tail); + r->sq_mask=(unsigned*)(sq+r->p.sq_off.ring_mask); + r->sq_entries=(unsigned*)(sq+r->p.sq_off.ring_entries); + r->sq_array=(unsigned*)(sq+r->p.sq_off.array); + r->cq_head=(unsigned*)(cq+r->p.cq_off.head); + r->cq_tail=(unsigned*)(cq+r->p.cq_off.tail); + r->cq_mask=(unsigned*)(cq+r->p.cq_off.ring_mask); + r->cq_entries=(unsigned*)(cq+r->p.cq_off.ring_entries); + r->sqes=(struct io_uring_sqe*)r->sqes_map; + r->cqes=(struct io_uring_cqe*)(cq+r->p.cq_off.cqes); + return 0; +} + +static inline int coli_uring_set_workers(ColiUring *r,unsigned workers){ + unsigned limits[2]={workers,workers}; /* bounded, unbounded io-wq workers */ + return (int)syscall(SYS_io_uring_register,r->fd,IORING_REGISTER_IOWQ_MAX_WORKERS,limits,2); +} + +static inline int coli_uring_prep_read(ColiUring *r,int fd,void *buf,size_t len, + int64_t off,uint64_t user_data){ + if(!len || len>UINT32_MAX){ errno=EINVAL; return -1; } + unsigned head=coli_uring_load_acquire(r->sq_head); + unsigned tail=__atomic_load_n(r->sq_tail,__ATOMIC_RELAXED); + if(tail-head>=*r->sq_entries){ errno=EAGAIN; return -1; } + unsigned idx=tail&*r->sq_mask; + struct io_uring_sqe *sqe=&r->sqes[idx]; + memset(sqe,0,sizeof(*sqe)); + sqe->opcode=IORING_OP_READ; + /* Cold regular-file reads are allowed to execute inline during + * io_uring_enter() unless forced async. That serializes the submitter on + * filesystems without native nonblocking buffered reads and destroys the + * intended I/O/compute overlap. io-wq gives the ring a real bounded worker + * pool while CQEs retain completion ordering/ownership here. */ + sqe->flags=IOSQE_ASYNC; + sqe->fd=fd; + sqe->off=(uint64_t)off; + sqe->addr=(uint64_t)(uintptr_t)buf; + sqe->len=(uint32_t)len; + sqe->user_data=user_data; + r->sq_array[idx]=idx; + coli_uring_store_release(r->sq_tail,tail+1); + return 0; +} + +static inline int coli_uring_enter(ColiUring *r,unsigned min_complete){ + for(;;){ + unsigned head=coli_uring_load_acquire(r->sq_head); + unsigned tail=coli_uring_load_acquire(r->sq_tail); + unsigned submit=tail-head; + unsigned flags=min_complete?IORING_ENTER_GETEVENTS:0; + int n=(int)syscall(SYS_io_uring_enter,r->fd,submit,min_complete,flags,NULL,0); + if(n>=0) return n; + if(errno!=EINTR) return -1; + } +} + +static inline int coli_uring_peek(ColiUring *r,struct io_uring_cqe *out){ + unsigned head=__atomic_load_n(r->cq_head,__ATOMIC_RELAXED); + unsigned tail=coli_uring_load_acquire(r->cq_tail); + if(head==tail) return 0; + *out=r->cqes[head&*r->cq_mask]; + coli_uring_store_release(r->cq_head,head+1); + return 1; +} + +#endif /* __linux__ */ +#endif /* COLI_URING_H */ diff --git a/docs/CACHE_ROUTE.md b/docs/CACHE_ROUTE.md new file mode 100644 index 0000000..78f47e9 --- /dev/null +++ b/docs/CACHE_ROUTE.md @@ -0,0 +1,65 @@ +# CACHE_ROUTE — opt-in cache-aware MoE routing + +**Default: OFF.** Stock full top-K router behavior unless `CACHE_ROUTE=1`. + +Paper-style max-rank selection (arXiv:2412.00099): keep true top-`J` always; +fill remaining slots preferring experts already **resident** (pin ∪ LRU) that +still rank inside top-`M`. + +This is **routing-side** (can change which experts run). Complementary to +**PILOT** (next-layer *prefetch* of weights; does not change expert IDs). + +## Flags + +| Env | Default | Meaning | +|-----|---------|---------| +| `CACHE_ROUTE` | `0` | `1` = max-rank cache-aware fill (pin∪LRU prefer) | +| `ROUTE_J` | `2` | Always take true top-J (even if uncached) | +| `ROUTE_M` | `12` | Max-rank window for resident preference | +| `ROUTE_P` | `0` | Optional cumulative mass window (`0` = use fixed M) | +| `ROUTE_ALPHA` | `1` | Scale gate mass of *substituted* experts before renorm (`1` = off) | +| `ROUTE_AGREE` | auto | Overlap% + KL vs true top-K; auto-on when `CACHE_ROUTE=1` | + +## One-liners + +```bash +# Stock full-K (leaderboard-comparable routing) +CACHE_ROUTE=0 ./coli chat + +# Experimental CACHE_ROUTE +CACHE_ROUTE=1 ROUTE_J=2 ROUTE_M=12 ./coli chat + +# Wider prefer window (more hit, more possible swap) +CACHE_ROUTE=1 ROUTE_J=2 ROUTE_M=16 ./coli chat +``` + +## Stats + +Footer / serve `STAT` when enabled: + +- `swap N%` / `swap_pct` — fraction of chosen slots not in true top-K +- `route_swaps` / `route_slots` — raw substitution counters +- `route_agree` — |chosen ∩ true top-K| / K +- `route_kl` — mass KL (true top-K vs chosen) +- `hit N%` — expert cache hit (disk residency) + +## A/B vs PILOT + +```bash +# A: cache-aware routing only +CACHE_ROUTE=1 PILOT=0 ... + +# B: lookahead prefetch only (does not change expert IDs) +CACHE_ROUTE=0 PILOT=1 ... + +# C: both +CACHE_ROUTE=1 PILOT=1 ... +``` + +## Scope of this PR + +**Routing-only** + telemetry + this note. Does **not** require CUDA/fuse/device-tier +patches — CPU streaming + pin/LRU is enough to A/B the lever against PILOT / #119. + +Treat as experimental until quality gates (e.g. `./coli bench`) pass; do not +default `CACHE_ROUTE=1`. diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.md index 6d73f3a..d345b7b 100644 --- a/docs/ENVIRONMENT.md +++ b/docs/ENVIRONMENT.md @@ -39,20 +39,27 @@ Format: `VAR` — default — effect. | `COLI_METAL_GEMM_MIN` | `16` | Minimum matmul rows to dispatch a GEMM to the GPU (below this, stays on CPU). | | `COLI_METAL_SPIN` | off | Keep a GPU keep-alive spinner running (reduces dispatch latency; costs power). | | `PIPE` | `0` (off) | Overlap expert disk-load with matmul via I/O worker threads. Byte-identical output; reorders I/O. `PIPE=1` opts in. | -| `PIPE_WORKERS` | `8` | Number of I/O worker threads when `PIPE=1`. Tune to your SSD (fewer avoids over-subscribing cores). | +| `PIPE_WORKERS` | `8` | Number of pthread loaders when `PIPE=1`, or the io-wq worker maximum per ring when `URING=1` (capped at 64). Tune to SSD queue depth and available cores. | +| `URING` | `0` (off) | Linux-only queued expert I/O. `URING=1` implies `PIPE=1`, forces cold reads through io-wq (`IOSQE_ASYNC`), replaces blocking loader pthreads and spin waits with batched SQEs/CQEs, and batches `PILOT_REAL` loads on a separate ring. Use `DIRECT=1` for cold NVMe to avoid page-cache copy/readahead limits. Fails clearly if the kernel denies io_uring; incompatible with `COLI_MMAP=1`. | | `DIRECT` | `0` (off) | Use `O_DIRECT`/unbuffered reads for expert slabs. Helps sustained NVMe; keeps the zero-copy GPU path. | | `COLI_NO_OMP_TUNE` | off | **Kill-switch** for the OpenMP hot-thread tuning (`OMP_WAIT_POLICY=active` spin + proc-bind). Set `=1` when the CPU is mostly waiting on the GPU (Metal) so spin doesn't steal the shared power budget. | | `MLOCK` | `-1` (auto: on for macOS) | Wire the streamed expert cache into physical RAM (`mlock`) to dodge the memory compressor. `0` off, `1` force. | | `CAP_RAISE` | `1` (on) | Let the engine raise the expert-cache cap above `topk` when RAM allows (bigger batches). `0` fixes the cap. | | `PREFETCH` | `0` | Prefetch depth for streamed experts. | | `COLI_MMAP` | `0` | `mmap` the weights instead of read()-ing into slabs. | -| `PIN` | unset | Path to a `.coli_usage` file; pins the hottest experts into a resident "hot store" at startup. | +| `PIN` | unset | Path to a `.coli_usage`/stats file; pins the hottest experts into a resident "hot store" at startup. **`PIN=auto`** seeds from the model dir's live `.coli_usage` (appended after every turn, so each restart's pin placement follows the accumulated real workload) with `stats.txt` as the fallback for a virgin model dir; neither present → no pin this run. | | `PIN_GB` | `10.0` | Size budget (GB) for the pinned hot store when `PIN` is set. | | `AUTOPIN` | `1` (on) | Auto-pin the hot store from usage history once ≥5000 selections are recorded. | | `REPIN` | `0` (off) | Live re-pin the hot store every N emitted tokens (RFC). | | `PILOT` | `0` (off) | Router-piloted cross-layer expert prefetch. | | `PILOT_REAL` | `0` (off) | Value-preserving real cross-layer prefetch loads (`PILOT_REAL=1` opts in). | | `PILOT_K` | `6` if `PILOT_REAL` else `8` | Number of experts the pilot prefetches per step. | +| `CACHE_ROUTE` | `0` (off) | Opt-in max-rank cache-aware MoE routing (pin∪LRU prefer within top-M). See [CACHE_ROUTE.md](CACHE_ROUTE.md). | +| `ROUTE_J` | `2` | Sacred top ranks always taken when `CACHE_ROUTE=1`. | +| `ROUTE_M` | `12` | Max-rank window for resident preference when `CACHE_ROUTE=1`. | +| `ROUTE_P` | `0` | Cumulative mass window for CACHE_ROUTE (`0` = fixed M). | +| `ROUTE_ALPHA` | `1` | Scale gate mass of substituted experts before renorm (`1` = off). | +| `ROUTE_AGREE` | auto | Overlap% + KL vs true top-K; auto-on when `CACHE_ROUTE=1`. | | `ABSORB` | `-1` (auto: absorbed for S≤4) | MLA attention absorption mode. | | `IDOT` | `1` | Integer dot-product kernel. `IDOT=0` uses exact f32 kernels (for A/B numerical checks). | | `COLI_POLICY` | `quality` | Resource policy: `quality`, `balanced`, or `experimental-fast`. | diff --git a/docs/media/colibri-brain.png b/docs/media/colibri-brain.png new file mode 100644 index 0000000..0cd6c90 Binary files /dev/null and b/docs/media/colibri-brain.png differ diff --git a/docs/media/colibri-dashboard.png b/docs/media/colibri-dashboard.png new file mode 100644 index 0000000..123fd3e Binary files /dev/null and b/docs/media/colibri-dashboard.png differ diff --git a/web/src/App.tsx b/web/src/App.tsx index 57e5f98..d4e0e41 100644 --- a/web/src/App.tsx +++ b/web/src/App.tsx @@ -34,7 +34,11 @@ import { Brain } from "./Brain" import { persistPublicSettings, stored } from "@/lib/storage" import { cn } from "@/lib/utils" -const message = (role: ChatMessage["role"], content: string): ChatMessage => ({ id: crypto.randomUUID(), role, content }) +const message = (role: ChatMessage["role"], content: string): ChatMessage => { + let id: string + try { id = crypto.randomUUID() } catch { id = 'xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx'.replace(/[xy]/g, c => { const r = Math.random() * 16 | 0; return (c === 'x' ? r : (r & 0x3 | 0x8)).toString(16) }) } + return { id, role, content } +} export default function App() { // When the page is served by the engine itself (coli web), same-origin is the diff --git a/web/src/Brain.tsx b/web/src/Brain.tsx index ac20695..6811238 100644 --- a/web/src/Brain.tsx +++ b/web/src/Brain.tsx @@ -4,6 +4,7 @@ import { BrainCircuit, Flame, Layers } from "lucide-react" import { endpoint } from "@/lib/api" interface ExpertMap { rows: number; cols: number; map: string; hits: string; seq: number } +interface AtlasEntry { affinity: Record; entropy: number; top: string; label: string } const TIER_NAME = ["Disk", "RAM", "VRAM"] const TIER_RGB: [number, number, number][] = [[58, 71, 80], [90, 155, 216], [78, 214, 165]] @@ -26,11 +27,19 @@ export function Brain({ baseUrl, apiKey, connected }: { baseUrl: string; apiKey: const wrapRef = useRef(null) const [wrapSize, setWrapSize] = useState({ w: 1200, h: 700 }) const [data, setData] = useState(null) + const [atlas, setAtlas] = useState | null>(null) const [tip, setTip] = useState<{ x: number; y: number; row: number; col: number; tier: number; heat: number } | null>(null) const pulseRef = useRef(null) // per-expert pulse intensity 0..1 const lastSeq = useRef(0) const rafRef = useRef(0) + // load the expert atlas if published (measured topic affinity, #175) + useEffect(() => { + fetch("/experts.json").then(r => r.ok ? r.json() : null).then(d => { + if (d?.experts) setAtlas(d.experts) + }).catch(() => {}) + }, []) + // track container size for responsive cell sizing useEffect(() => { const el = wrapRef.current @@ -146,14 +155,26 @@ export function Brain({ baseUrl, apiKey, connected }: { baseUrl: string; apiKey: setTip(null)} /> {!connected &&

Connect to the engine to see the cortex.

} - {tip && data && ( + {tip && data && (() => { + const isMtp = tip.row === data.rows - 1 + const realLayer = isMtp ? 78 : tip.row + 3 + const entry = atlas?.[`${realLayer}:${tip.col}`] + return (
-
Layer row {tip.row}{tip.row === data.rows - 1 ? " (MTP)" : ""} · Expert {tip.col}
+
Layer {realLayer}{isMtp ? " (MTP)" : ""} · Expert {tip.col}
Tier: {TIER_NAME[tip.tier]}
Heat: {tip.heat === 0 ? "never routed" : `~2^${tip.heat} selections`}
-
{depthRole(tip.row, data.rows, tip.row === data.rows - 1)}
+ {entry ? <> +
+ {entry.label.startsWith("specialist") ? `⭐ Specialist: ${entry.top}` : "Generalist"} + (entropy {entry.entropy}) +
+
{Object.entries(entry.affinity).sort((a, b) => b[1] - a[1]).slice(0, 3) + .map(([c, p]) => `${c} ${Math.round(p * 100)}%`).join(" · ")}
+ :
{depthRole(tip.row, data.rows, isMtp)}
}
- )} + ) + })()} ) } diff --git a/web/src/index.css b/web/src/index.css index 59d5bc6..b428805 100644 --- a/web/src/index.css +++ b/web/src/index.css @@ -146,3 +146,8 @@ button:focus-visible, input:focus-visible, textarea:focus-visible, select:focus- .brain-legend { gap: 8px; font-size: 10px; } .brain-tip { max-width: 220px; font-size: 10px; } } + +/* atlas hover extras */ +.brain-tip-spec { color: var(--primary); font-weight: 700; } +.brain-tip-spec small, .brain-tip-aff { color: #8b9aa3; font-weight: 400; } +.brain-tip-aff { font-size: 10px; }