Nicholas Beerbower
9e2d41567c
tests/tok_o200k_tiny.json is a synthetic byte-level BPE (274 vocab, a
few KB) whose Split regex is the o200k pattern; expected ids in
tok_o200k_cases.txt were generated by HF tokenizers on that same file.
test_tok_o200k (in TEST_BINS) scores 40/40 encode + 40/40 round-trip:
case-transition splits, contractions, digit groups, the [\r\n/]* tail,
whitespace branches, CJK/Greek/Cyrillic, added-token atomicity.
The cl100k path is untouched by construction — dispatch requires
\p{Lu} in the tokenizer's own Split pattern, which cl100k lacks — and
stays covered by the GLM oracle (verified on this branch: 32/32).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
41 lines
1.3 KiB
Plaintext
41 lines
1.3 KiB
Plaintext
hello world 259,32,119,111,114,263
|
|
HelloWorld 72,101,257,111,262
|
|
XMLHttpRequest 88,77,76,72,116,116,112,82,101,113,117,101,115,116
|
|
helloWORLDhello 259,87,79,82,76,68,259
|
|
dog's 100,111,103,270
|
|
DON'T 68,79,78,39,84
|
|
don't 100,111,110,39,116
|
|
I'll've 73,39,257,39,118,101
|
|
O'Brien 79,39,66,114,105,101,110
|
|
the theatre 265,32,116,256,97,116,114,101
|
|
12345 268,52,53
|
|
a1b22c333d4444 97,49,98,50,50,99,51,51,51,100,52,52,52,52
|
|
3.14 51,46,49,52
|
|
http://x.com/a/b 104,116,116,112,58,47,47,120,46,99,111,109,47,97,47,98
|
|
path/to/file 112,97,264,47,116,111,47,102,105,108,101
|
|
a//b///c 97,47,47,98,47,47,47,99
|
|
!!\r\n//x 33,33,13,10,47,47,120
|
|
one\ntwo\r\nthree 111,110,101,10,116,119,111,13,10,264,114,101,101
|
|
\n x 32,32,10,32,32,120
|
|
32,32,32
|
|
a b c 97,32,32,98,32,32,32,99
|
|
tab\there 116,269,9,256,114,101
|
|
Café 67,97,102,101,204,129
|
|
naiveBayes 110,97,105,118,101,66,97,121,101,115
|
|
Éclair 195,137,99,108,97,105,114
|
|
北京大学 229,140,151,228,186,172,229,164,167,229,173,166
|
|
ΑΒαβ 206,145,206,146,206,177,206,178
|
|
Иван 208,152,208,178,208,176,208,189
|
|
ẞßscharf 225,186,158,195,159,115,99,104,97,114,102
|
|
i̇stanbul 105,204,135,115,116,97,110,98,117,108
|
|
hello<|endoftext|>world 259,274,119,111,114,263
|
|
<|message_user|>hi 275,104,105
|
|
mixedCASEand123 109,105,120,101,100,67,65,83,69,97,110,100,268
|
|
's 270
|
|
's 32,39,115
|
|
A 65
|
|
aB 97,66
|
|
Ab 65,98
|
|
AB 65,66
|
|
ab 269
|