A model reads tokens, up to a limit.
Five pieces of Larkspur Cloud text, tokenized by GPT-2 and by all-MiniLM-L6-v2 exactly as transformers does it. Move the token limit: everything after the magenta rule is cut off before the model sees it, and nothing warns you.
- Tokens in the text
- Tokens the model reads
- Tokens cut off
- Characters per token
What the model receives
Shaded runs alternate, one per token. Hover a token for its id.
Same text, two tokenizers
| Text | Characters | GPT-2 | MiniLM |
|---|
Token counts as each model receives them; MiniLM’s include [CLS] and [SEP]. M10 estimates tokens as characters divided by four.