Dennis Paul
a28f31fa3b
Probe sweep + affinity analysis + leave-one-prompt-out validation, so anyone can build and
cross-validate the atlas on their own box rather than trusting one machine.
The four traps this harness exists to control (each silently corrupts the atlas):
--topp prunes experts by cumulative probability. Measured, same prompt:
topp=0 -> 21,000 selections across 7,587 distinct experts
topp=0.7 -> 11,944 selections across 4,687 distinct experts
It hides 38% of the experts, and it is the recommended speed setting.
MTP/DRAFT eusage is incremented inside moe(), BEFORE verification, so rejected
speculative drafts count experts routed for text never emitted.
.coli_usage is loaded at startup and accumulates, so a naive STATS dump contains all
prior history rather than this run.
autocorrelation: routing within one run is highly correlated, so an expert firing 38
times during one prompt is ONE observation, not 38. Entropy/chi-square on raw
selections certifies single-prompt flukes as perfect specialists — analyze.py
therefore requires affinity to replicate across a category independent prompts.
Result on GLM-5.2 744B int4 (Zen5, CPU routing path), 10 topics x 3 prompts x 64 tokens:
leave-one-prompt-out accuracy 29/30 = 96.7% (chance 10%)
strong specialists (spec>=0.5) 1,041 / 13,260 (7.9%)
specialisation vs depth layer 3 ~0.07 -> layers 18-58 ~0.19-0.27
replication gate rejected 587 single-prompt flukes
The one miss is the interesting part: a Chinese-language poetry prompt classifies as poetry,
not Chinese — routing follows the task over the language.
55 lines
2.7 KiB
JSON
55 lines
2.7 KiB
JSON
{
|
|
"_comment": "Probe set for the GLM-5.2 Expert Atlas (#175). 10 categories x 3 prompts. Prompts are deliberately varied in phrasing within a category so the affinity vector reflects the TOPIC, not one prompt's surface form. Lengths are kept within a narrow band across categories: prefill routes the prompt tokens too, so a long prompt in one category would inflate its counts.",
|
|
|
|
"code_python": [
|
|
"Write a Python function that merges two sorted lists into one sorted list.",
|
|
"In Python, explain the difference between a generator and a list comprehension.",
|
|
"Refactor this Python snippet to avoid a nested loop: for a in xs: for b in ys: if a==b: out.append(a)"
|
|
],
|
|
"code_sql": [
|
|
"Write a SQL query that returns the top 5 customers by total order value.",
|
|
"Explain what a LEFT JOIN does differently from an INNER JOIN in SQL.",
|
|
"Write SQL to add an index on the email column of a users table and explain when it helps."
|
|
],
|
|
"math_proof": [
|
|
"Prove that the square root of 2 is irrational.",
|
|
"Show that the sum of the first n odd numbers equals n squared.",
|
|
"Explain why the derivative of e^x is e^x."
|
|
],
|
|
"chinese": [
|
|
"请用三句话解释什么是机器学习。",
|
|
"写一首关于秋天的短诗。",
|
|
"中国的四大发明是什么？请简要说明。"
|
|
],
|
|
"german": [
|
|
"Erkläre in drei Sätzen, wie ein Verbrennungsmotor funktioniert.",
|
|
"Schreibe eine kurze formelle E-Mail, in der du einen Termin absagst.",
|
|
"Was ist der Unterschied zwischen Dativ und Akkusativ im Deutschen?"
|
|
],
|
|
"poetry": [
|
|
"Write a short poem about the sea at night.",
|
|
"Compose four lines of verse about an empty train station.",
|
|
"Write a haiku about the first snow of winter."
|
|
],
|
|
"law": [
|
|
"Explain in plain terms what consideration means in contract law.",
|
|
"What is the difference between a misdemeanor and a felony?",
|
|
"Summarize what the term 'force majeure' covers in a commercial contract."
|
|
],
|
|
"medicine": [
|
|
"Explain the difference between type 1 and type 2 diabetes.",
|
|
"What are the common symptoms of iron deficiency anemia?",
|
|
"Describe how a vaccine produces immunity."
|
|
],
|
|
"json_format": [
|
|
"Return a JSON object with keys name, age, and email for a fictional user. Output only JSON.",
|
|
"Convert this to JSON: name Alice, roles admin and editor, active true. Output only JSON.",
|
|
"Write a JSON schema for an object with a required string field 'id' and an optional integer 'count'."
|
|
],
|
|
"casual_chat": [
|
|
"Hey, what's a good way to spend a rainy Sunday afternoon?",
|
|
"I'm feeling pretty tired today. Any tips to get through the afternoon?",
|
|
"What's your favourite kind of weather, and why?"
|
|
]
|
|
}
|