Files

T

Dennis Paul a28f31fa3b tools: expert_atlas — confound-controlled probe harness for the GLM-5.2 expert atlas (#175 ) (#218 )

Probe sweep + affinity analysis + leave-one-prompt-out validation, so anyone can build and
cross-validate the atlas on their own box rather than trusting one machine.

The four traps this harness exists to control (each silently corrupts the atlas):

  --topp     prunes experts by cumulative probability. Measured, same prompt:
             topp=0   -> 21,000 selections across 7,587 distinct experts
             topp=0.7 -> 11,944 selections across 4,687 distinct experts
             It hides 38% of the experts, and it is the recommended speed setting.
  MTP/DRAFT  eusage is incremented inside moe(), BEFORE verification, so rejected
             speculative drafts count experts routed for text never emitted.
  .coli_usage is loaded at startup and accumulates, so a naive STATS dump contains all
             prior history rather than this run.
  autocorrelation: routing within one run is highly correlated, so an expert firing 38
             times during one prompt is ONE observation, not 38. Entropy/chi-square on raw
             selections certifies single-prompt flukes as perfect specialists — analyze.py
             therefore requires affinity to replicate across a category independent prompts.

Result on GLM-5.2 744B int4 (Zen5, CPU routing path), 10 topics x 3 prompts x 64 tokens:

  leave-one-prompt-out accuracy   29/30 = 96.7%   (chance 10%)
  strong specialists (spec>=0.5)  1,041 / 13,260  (7.9%)
  specialisation vs depth         layer 3 ~0.07 -> layers 18-58 ~0.19-0.27
  replication gate rejected       587 single-prompt flukes

The one miss is the interesting part: a Chinese-language poetry prompt classifies as poetry,
not Chinese — routing follows the task over the language.

2026-07-15 07:41:24 +02:00

3.8 KiB

Raw Blame History

Expert Atlas — what does each of the 19,456 experts actually do?

Probe harness for #175. Runs a set of topic-tagged prompts, dumps each run's expert-routing histogram, and turns them into a per-expert topic-affinity vector.

cd c
export COLI_MODEL=/path/to/glm52_i4
./tools/expert_atlas/sweep.sh                             # 30 probes (10 topics x 3 prompts)
python3 tools/expert_atlas/analyze.py  --stats atlas_out/stats --out atlas_out/experts.json
python3 tools/expert_atlas/validate.py atlas_out/stats 200 # leave-one-prompt-out check

Read this before you trust any atlas

Four things silently corrupt this measurement. The sweep script controls all of them; if you roll your own, don't skip them.

trap	effect	control
`--topp`	prunes experts by cumulative probability — measured: it hides 38% of the distinct experts (7,587 → 4,687). It is also the recommended speed setting.	`TOPP=0`
speculative drafts	`eusage` is incremented inside `moe()`, before verification, so rejected drafts count. Those are experts routed for text the model never emitted.	`MTP=0 DRAFT=0`
`.coli_usage`	is loaded at startup and accumulates, so a naive `STATS` dump contains all prior history, not this run.	remove per run (script backs it up and restores)
autocorrelation	routing inside one run is highly correlated — the same context routes to the same experts token after token. An expert firing 38 times during one prompt is one observation, not 38. Chi-square/entropy on raw selections will certify single-prompt flukes as perfect specialists.	`analyze.py` requires the affinity to replicate across a category's independent prompts

The CUDA expert tier is also not run-to-run deterministic, so the sweep uses --gpu none. Tier config only decides where weights live, not what the router picks, so this costs nothing.

Method

analyze.py:

n[e][c] — selections of expert e while running category c
f[e][c] = n[e][c] / N[c] — normalise by category size (prefill routes the prompt too, so a verbose category would otherwise look busier)
p(c|e) — renormalise into a topic distribution per expert, i.e. base-rate corrected. Ranking by raw count instead just rediscovers which experts are popular in general.
spec(e) = 1 − H(p(c|e)) / log C — 0 = generalist, 1 = fires for exactly one topic
replication gate — an expert is only a candidate specialist for c if it fires in ≥2 of c's independent prompts

validate.py — leave-one-prompt-out. Build each category's top-K specialist set from its other prompts, then check which set the held-out prompt's routing actually lands in. If specialisation were an artifact of prompt wording, the held-out prompt would not prefer its own category.

Result on GLM-5.2 744B int4 (Zen5, CPU routing path)

Leave-one-prompt-out: 29/30 = 96.7% (chance 10%). Specialisation is a property of the topic, not of prompt wording.
The single miss is instructive: 写一首关于秋天的短诗 ("write a short poem about autumn") is classified poetry, not Chinese — routing follows the task over the language.
Only 7.9% of experts are strong specialists (spec ≥ 0.5). The "one expert = one topic" picture is wrong for ~92% of them.
Specialisation rises with depth: layer 3 ≈ 0.07 (generalist, token/syntax level) → layers 18–58 ≈ 0.19–0.27.
The replication gate removed 587 experts that looked like flawless specialists on one prompt.

Extending it

probes.json is the whole probe set — add categories and prompts. Use 3+ prompts per category with varied phrasing, or the replication gate has nothing to check and you are back to measuring one prompt. Keep prompt lengths in a similar band across categories.

3.8 KiB Raw Blame History Unescape Escape