Dennis Paul
a28f31fa3b
Probe sweep + affinity analysis + leave-one-prompt-out validation, so anyone can build and
cross-validate the atlas on their own box rather than trusting one machine.
The four traps this harness exists to control (each silently corrupts the atlas):
--topp prunes experts by cumulative probability. Measured, same prompt:
topp=0 -> 21,000 selections across 7,587 distinct experts
topp=0.7 -> 11,944 selections across 4,687 distinct experts
It hides 38% of the experts, and it is the recommended speed setting.
MTP/DRAFT eusage is incremented inside moe(), BEFORE verification, so rejected
speculative drafts count experts routed for text never emitted.
.coli_usage is loaded at startup and accumulates, so a naive STATS dump contains all
prior history rather than this run.
autocorrelation: routing within one run is highly correlated, so an expert firing 38
times during one prompt is ONE observation, not 38. Entropy/chi-square on raw
selections certifies single-prompt flukes as perfect specialists — analyze.py
therefore requires affinity to replicate across a category independent prompts.
Result on GLM-5.2 744B int4 (Zen5, CPU routing path), 10 topics x 3 prompts x 64 tokens:
leave-one-prompt-out accuracy 29/30 = 96.7% (chance 10%)
strong specialists (spec>=0.5) 1,041 / 13,260 (7.9%)
specialisation vs depth layer 3 ~0.07 -> layers 18-58 ~0.19-0.27
replication gate rejected 587 single-prompt flukes
The one miss is the interesting part: a Chinese-language poetry prompt classifies as poetry,
not Chinese — routing follows the task over the language.
83 lines
3.7 KiB
Bash
Executable File
83 lines
3.7 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# GLM-5.2 Expert Atlas — probe sweep (#175).
|
|
#
|
|
# cd c && ./tools/expert_atlas/sweep.sh [probes.json] [outdir]
|
|
#
|
|
# Env: COLI_MODEL, NGEN (default 64), COLI (default ./coli)
|
|
#
|
|
# ---------------------------------------------------------------------------------------------
|
|
# THE CONFOUNDS THIS SCRIPT EXISTS TO CONTROL. Each one silently corrupts the atlas.
|
|
#
|
|
# TOPP=0 --topp prunes experts by cumulative probability. Measured on GLM-5.2 int4,
|
|
# same prompt, only top-p changed:
|
|
# topp=0 -> 21,000 selections across 7,587 distinct experts
|
|
# topp=0.7 -> 11,944 selections across 4,687 distinct experts
|
|
# It hides 38% of the experts. It is also the RECOMMENDED SPEED SETTING, so this
|
|
# is very easy to walk into: with top-p on you profile the pruner, not the model.
|
|
#
|
|
# MTP=0 Speculative drafts route experts for tokens that are later REJECTED and never
|
|
# DRAFT=0 emitted (eusage is incremented inside moe(), before verification). Those counts
|
|
# would describe text the model never produced.
|
|
#
|
|
# --gpu none The CUDA expert tier is not run-to-run deterministic (VRAM placement shifts
|
|
# between runs). Routing on the CPU path is reproducible. The tier only decides
|
|
# where weights live, not what the router picks, so CPU costs nothing here.
|
|
#
|
|
# --temp 0 Greedy: deterministic continuation, reproducible atlas.
|
|
#
|
|
# rm .coli_usage before EVERY run
|
|
# eusage is LOADED from <model>/.coli_usage at startup and written back at exit, so
|
|
# a naive STATS dump contains ALL PRIOR HISTORY, not this run. Removing it per run
|
|
# makes each dump exactly one probe. The user's learned cache is backed up and
|
|
# restored on exit.
|
|
#
|
|
# Prompt lengths are kept in a narrow band across categories: prefill routes the prompt tokens
|
|
# too, so a verbose category would otherwise simply look "busier".
|
|
# ---------------------------------------------------------------------------------------------
|
|
set -uo pipefail
|
|
HERE="$(cd "$(dirname "$0")" && pwd)"
|
|
PROBES="${1:-$HERE/probes.json}"
|
|
OUT="${2:-./atlas_out}"
|
|
COLI="${COLI:-./coli}"
|
|
NGEN="${NGEN:-64}"
|
|
MODEL="${COLI_MODEL:?set COLI_MODEL to the snapshot directory}"
|
|
|
|
[ -f "$PROBES" ] || { echo "no probe file: $PROBES" >&2; exit 1; }
|
|
[ -x "$COLI" ] || { echo "no coli at $COLI (run from c/, or set COLI=)" >&2; exit 1; }
|
|
mkdir -p "$OUT/stats"
|
|
|
|
export COLI_MODEL="$MODEL" MTP=0 DRAFT=0 TOPP=0
|
|
|
|
USAGE="$MODEL/.coli_usage"
|
|
BACKUP="$OUT/.coli_usage.backup"
|
|
[ -f "$USAGE" ] && cp "$USAGE" "$BACKUP" && echo "backed up $USAGE"
|
|
restore(){ [ -f "$BACKUP" ] && cp "$BACKUP" "$USAGE" && echo "restored $USAGE"; }
|
|
trap restore EXIT
|
|
|
|
python3 - "$PROBES" > "$OUT/runlist.tsv" <<'PY'
|
|
import json, sys
|
|
for cat, prompts in json.load(open(sys.argv[1])).items():
|
|
if cat.startswith('_'):
|
|
continue
|
|
for i, p in enumerate(prompts):
|
|
print(f"{cat}\t{i}\t{p}")
|
|
PY
|
|
|
|
n=$(wc -l < "$OUT/runlist.tsv"); i=0
|
|
echo "$n probes -> $OUT/stats"
|
|
while IFS=$'\t' read -r cat idx prompt; do
|
|
i=$((i+1))
|
|
dst="$OUT/stats/${cat}_${idx}.txt"
|
|
[ -s "$dst" ] && { echo " [$i/$n] $cat/$idx (cached)"; continue; }
|
|
rm -f "$USAGE" # start from an EMPTY routing history
|
|
STATS="$dst" "$COLI" run "$prompt" --ngen "$NGEN" --ctx 4096 --gpu none --temp 0 \
|
|
> "$OUT/stats/${cat}_${idx}.log" 2>&1
|
|
echo " [$i/$n] $cat/$idx $(grep -aoE '[0-9]+ selections across [0-9]+ distinct experts' \
|
|
"$OUT/stats/${cat}_${idx}.log" | head -1)"
|
|
done < "$OUT/runlist.tsv"
|
|
|
|
echo
|
|
echo "next:"
|
|
echo " python3 $HERE/analyze.py --stats $OUT/stats --out $OUT/experts.json"
|
|
echo " python3 $HERE/validate.py $OUT/stats 200"
|