Tokenizer playground
i
Tokenizer playground
A tokenizer splits text into the small pieces a language model can read. A piece may be a word, part of a word, punctuation, or a space.
Edit the sentence, then inspect the model-readable pieces hiding beneath the text.
Characters56
Tokens10
Chars / token5.6
Whitespace often becomes part of the next token. Click any piece to inspect its deterministic teaching ID and bytes.
Selected pieceTokenization
- Token ID
i
Token IDThe vocabulary number assigned to a token. The model receives this number rather than the visible text.
- 1063
- UTF-8 bytes
i
UTF-8 bytesThe raw numbers computers use to store text characters. UTF-8 is the common encoding used on the web.
- 84 111 107 101 110 105 122 97 116 105 111 110
- Characters
- 12