tools: expert_atlas — confound-controlled probe harness for the GLM-5.2 expert atlas (#175) (#218)

Probe sweep + affinity analysis + leave-one-prompt-out validation, so anyone can build and
cross-validate the atlas on their own box rather than trusting one machine.

The four traps this harness exists to control (each silently corrupts the atlas):

  --topp     prunes experts by cumulative probability. Measured, same prompt:
             topp=0   -> 21,000 selections across 7,587 distinct experts
             topp=0.7 -> 11,944 selections across 4,687 distinct experts
             It hides 38% of the experts, and it is the recommended speed setting.
  MTP/DRAFT  eusage is incremented inside moe(), BEFORE verification, so rejected
             speculative drafts count experts routed for text never emitted.
  .coli_usage is loaded at startup and accumulates, so a naive STATS dump contains all
             prior history rather than this run.
  autocorrelation: routing within one run is highly correlated, so an expert firing 38
             times during one prompt is ONE observation, not 38. Entropy/chi-square on raw
             selections certifies single-prompt flukes as perfect specialists — analyze.py
             therefore requires affinity to replicate across a category independent prompts.

Result on GLM-5.2 744B int4 (Zen5, CPU routing path), 10 topics x 3 prompts x 64 tokens:

  leave-one-prompt-out accuracy   29/30 = 96.7%   (chance 10%)
  strong specialists (spec>=0.5)  1,041 / 13,260  (7.9%)
  specialisation vs depth         layer 3 ~0.07 -> layers 18-58 ~0.19-0.27
  replication gate rejected       587 single-prompt flukes

The one miss is the interesting part: a Chinese-language poetry prompt classifies as poetry,
not Chinese — routing follows the task over the language.

c/tools/expert_atlas/validate.py

@@ -0,0 +1,79 @@
+#!/usr/bin/env python3
+"""Leave-one-prompt-out validation of the Expert Atlas (#175).
+
+"Replicates across the 3 prompts I picked" is not the same as "generalises". The atlas is
+only real if a specialist set learned from SOME prompts predicts routing on a prompt it has
+never seen.
+
+Protocol, for every category c and every held-out prompt h of c:
+  1. build c's top-K specialist set from c's OTHER prompts only (h excluded entirely)
+  2. do the same for all other categories (using ALL their prompts — they never saw h either)
+  3. on the held-out run h, measure what share of routing selections land in each set
+  4. the atlas works if c's own set wins on h
+
+If specialisation were an artifact of prompt wording, the held-out prompt would not prefer
+its own category's set. Chance is 1/C.
+"""
+import glob, json, os, sys
+from collections import defaultdict
+
+STATS = sys.argv[1] if len(sys.argv) > 1 else "stats"
+K = int(sys.argv[2]) if len(sys.argv) > 2 else 200      # specialists per category
+
+runs, tot = {}, {}
+for path in sorted(glob.glob(os.path.join(STATS, "*.txt"))):
+    cat, idx = os.path.basename(path)[:-4].rsplit("_", 1)
+    d = {}
+    t = 0
+    for line in open(path):
+        p = line.split()
+        if len(p) == 3:
+            d[(int(p[0]), int(p[1]))] = int(p[2])
+            t += int(p[2])
+    runs[(cat, idx)] = d
+    tot[(cat, idx)] = t
+
+cats = sorted({c for c, _ in runs})
+idxs = {c: sorted(i for cc, i in runs if cc == c) for c in cats}
+C = len(cats)
+
+
+def specialists(cat, exclude):
+    """Top-K experts by lift for `cat`, computed WITHOUT the excluded run."""
+    share = defaultdict(lambda: defaultdict(float))
+    for c in cats:
+        used = [i for i in idxs[c] if not (c == cat and i == exclude)]
+        for i in used:
+            for k, n in runs[(c, i)].items():
+                share[k][c] += n / max(1, tot[(c, i)]) / len(used)
+    scored = []
+    for k, per in share.items():
+        s = sum(per.values())
+        if s <= 0:
+            continue
+        p = per[cat] / s
+        if per[cat] > 0:
+            scored.append((p, k))
+    scored.sort(reverse=True)
+    return {k for _, k in scored[:K]}
+
+
+print(f"leave-one-prompt-out, {C} categories, top-{K} specialists per category")
+print(f"chance = {100.0/C:.1f}%\n")
+hits = 0
+trials = 0
+for c in cats:
+    for h in idxs[c]:
+        sets = {cc: specialists(cc, h if cc == c else None) for cc in cats}
+        held = runs[(c, h)]
+        htot = max(1, tot[(c, h)])
+        scores = {cc: sum(held.get(k, 0) for k in sets[cc]) / htot for cc in cats}
+        win = max(scores, key=scores.get)
+        ok = win == c
+        hits += ok
+        trials += 1
+        own = 100 * scores[c]
+        best_other = 100 * max(v for cc, v in scores.items() if cc != c)
+        print(f"  {c:<12} prompt {h}  own-set {own:5.2f}%  best-other {best_other:5.2f}%  "
+              f"-> {'HIT ' if ok else 'MISS'} (predicted {win})")
+print(f"\naccuracy: {hits}/{trials} = {100*hits/trials:.1f}%   (chance {100.0/C:.1f}%)")