Vocabulary
The set of unique tokens known to a language model or NLP system.
Description
In natural language processing, vocabulary refers to the set of unique tokens that a model or system recognizes. This can include words, subwords, or characters, depending on the tokenization method used. The vocabulary is typically built from the training data and has a significant impact on the model's ability to understand and generate text. The size and composition of the vocabulary can affect model performance, memory usage, and the ability to handle out-of-vocabulary words.
Examples
- 📚 Word-level vocabulary
- 🧩 Subword vocabulary (e.g., in BERT or GPT models)
- 🔤 Character-level vocabulary
Applications
Related Terms
Featured
Animon AI
Create anime videos for free
Free AI PDF Reader
Free AI PDF Reader – Smarter Way to Understand Any PDF
AI Text Summarizer
AI Text Summarizer That Rocks: Faster Content Analysis
Google Nano Banana
Fast multimodal Gemini model for production
Ask AI Questions Online
Ask AI Questions for Free – Smart, Fast, and Human-Like Answers
ChatGPT Atlas
The browser with ChatGPT built in
Neurona AI Image Creator
AI image generator; AI art generator; face swap AI
Wan AI
Generate cinematic videos from text, image, and speech
Tidio
Smart, human-like support powered by AI — available 24/7.
AI Book Summarizer
AI Book Summarizer That Makes Books Easy to Grasp
Higgsfield AI
Cinematic AI video generator with pro VFX control
Sora 2
Transform Ideas into Stunning Videos with Sora 2
Abacus AI
The World's First Super Assistant for Professionals and Enterprises
Free AI Article Summarizer
Free Article Summarizer
Blackbox AI
Accelerate development with Blackbox AI's multi-model platform
Kimi AI
Kimi AI - K2 chatbot for long-context coding and research
