tests: o200k tokenizer coverage, no model download

tests/tok_o200k_tiny.json is a synthetic byte-level BPE (274 vocab, a
few KB) whose Split regex is the o200k pattern; expected ids in
tok_o200k_cases.txt were generated by HF tokenizers on that same file.
test_tok_o200k (in TEST_BINS) scores 40/40 encode + 40/40 round-trip:
case-transition splits, contractions, digit groups, the [\r\n/]* tail,
whitespace branches, CJK/Greek/Cyrillic, added-token atomicity.

The cl100k path is untouched by construction — dispatch requires
\p{Lu} in the tokenizer's own Split pattern, which cl100k lacks — and
stays covered by the GLM oracle (verified on this branch: 32/32).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Nicholas Beerbower
2026-07-16 15:17:21 -04:00
parent 364b9741e2
commit 9e2d41567c
4 changed files with 106 additions and 1 deletions
+40
View File
@@ -0,0 +1,40 @@
hello world 259,32,119,111,114,263
HelloWorld 72,101,257,111,262
XMLHttpRequest 88,77,76,72,116,116,112,82,101,113,117,101,115,116
helloWORLDhello 259,87,79,82,76,68,259
dog's 100,111,103,270
DON'T 68,79,78,39,84
don't 100,111,110,39,116
I'll've 73,39,257,39,118,101
O'Brien 79,39,66,114,105,101,110
the theatre 265,32,116,256,97,116,114,101
12345 268,52,53
a1b22c333d4444 97,49,98,50,50,99,51,51,51,100,52,52,52,52
3.14 51,46,49,52
http://x.com/a/b 104,116,116,112,58,47,47,120,46,99,111,109,47,97,47,98
path/to/file 112,97,264,47,116,111,47,102,105,108,101
a//b///c 97,47,47,98,47,47,47,99
!!\r\n//x 33,33,13,10,47,47,120
one\ntwo\r\nthree 111,110,101,10,116,119,111,13,10,264,114,101,101
\n x 32,32,10,32,32,120
32,32,32
a b c 97,32,32,98,32,32,32,99
tab\there 116,269,9,256,114,101
Café 67,97,102,101,204,129
naiveBayes 110,97,105,118,101,66,97,121,101,115
Éclair 195,137,99,108,97,105,114
北京大学 229,140,151,228,186,172,229,164,167,229,173,166
ΑΒαβ 206,145,206,146,206,177,206,178
Иван 208,152,208,178,208,176,208,189
ẞßscharf 225,186,158,195,159,115,99,104,97,114,102
i̇stanbul 105,204,135,115,116,97,110,98,117,108
hello<|endoftext|>world 259,274,119,111,114,263
<|message_user|>hi 275,104,105
mixedCASEand123 109,105,120,101,100,67,65,83,69,97,110,100,268
's 270
's 32,39,115
A 65
aB 97,66
Ab 65,98
AB 65,66
ab 269