Vocabulary
The set of unique tokens known to a language model or NLP system.
Description
In natural language processing, vocabulary refers to the set of unique tokens that a model or system recognizes. This can include words, subwords, or characters, depending on the tokenization method used. The vocabulary is typically built from the training data and has a significant impact on the model's ability to understand and generate text. The size and composition of the vocabulary can affect model performance, memory usage, and the ability to handle out-of-vocabulary words.
Examples
- 📚 Word-level vocabulary
- 🧩 Subword vocabulary (e.g., in BERT or GPT models)
- 🔤 Character-level vocabulary
Applications
Related Terms
Featured
ChatGPT Atlas
The browser with ChatGPT built in
Google Nano Banana
Fast multimodal Gemini model for production
Wan AI
Generate cinematic videos from text, image, and speech
Sora 2
Transform Ideas into Stunning Videos with Sora 2
AI Hairstyle
AI Hairstyle
AI Text Summarizer
AI Text Summarizer That Rocks: Faster Content Analysis
Free AI PDF Reader
Free AI PDF Reader – Smarter Way to Understand Any PDF
Blackbox AI
Accelerate development with Blackbox AI's multi-model platform
Abacus AI
The World's First Super Assistant for Professionals and Enterprises
Tidio
Smart, human-like support powered by AI — available 24/7.
AI Clothes Changer
AI Clothes Changer
Video Background Remover
AI Design
Higgsfield AI
Cinematic AI video generator with pro VFX control
AI Book Summarizer
AI Book Summarizer That Makes Books Easy to Grasp
Neurona AI Image Creator
AI image generator; AI art generator; face swap AI
Free AI Article Summarizer
Free Article Summarizer
Ask AI Questions Online
Ask AI Questions for Free – Smart, Fast, and Human-Like Answers
