AI glossary · Generative AI and LLMs
Token
A token is a small chunk of text, often a word or part of a word, that a language model reads and writes. Usage limits and prices are counted in tokens.
Also called tokenization, tokenisation
In more detail
Before a model can read text, a tokenizer splits it into tokens. Common words are usually one token; rare or long words are split into pieces. In English, one token is roughly three quarters of a word on average.
Tokens matter to everyday users because the context window, max tokens setting and API prices are all measured in them.
Example
The word “unbelievable” might be split into “un”, “believ” and “able”: three tokens.
Related terms
Context windowThe context window is how much text an AI model can consider at once, including your messages, files and its own replies, measured in tokens.Max tokensMax tokens is a setting that caps how long an AI’s reply can be, measured in tokens. When the limit is reached, the answer stops, even mid-sentence.Large language model (LLM)A large language model (LLM) is an AI model trained on vast amounts of text to understand and generate language. It powers chatbots like ChatGPT.APIAn API (application programming interface) lets one program use another’s features. AI companies offer APIs so developers can build their models into apps.
Browse related tools
AI tools listed on BroUseAI in these categories.