Tokenization

The process of splitting text into tokens — the fundamental units an LLM processes. Different models use different tokenizers, so the same text can produce different token counts. "Strawberry" might be 1 token or 3, depending on the tokenizer.