What Are Tokens?
Every LLM interaction has a finite context window — a fixed budget of tokens for both input and output. More context is not automatically better: it increases cost, latency, and hallucination risk. The skill is choosing the right context, not the most context.
You type a prompt and get a response. But before the model processes anything, your text is split into tokens — the atomic units of LLM processing. Every token counts against a fixed budget called the context window. Understanding tokens and context windows is essential for anyone building AI applications.
Table of Contents
- What Are Tokens?
- How Tokenization Works
- Token Counting
- The Context Window
- Context Budget
- Model Comparison
- Why More Context Is Not Better
- Tokens in Coding
- Tokens with Documents
- Tokens in RAG
- Tokens in AI Agents
- Context Management Strategies
- Cost Implications
- FAQ
- Conclusion
1. What Are Tokens?
A token is a piece of text that the model processes as a unit. Tokens are NOT the same as words:
"Hello, world!" → ["Hello", ",", " world", "!"] → 4 tokens
"unhappiness" → ["un", "happiness"] → 2 tokens
"GPT-4" → ["GPT", "-", "4"] → 3 tokens
"def calc_avg(n):" → ["def", " calc", "_avg", "(", "n", ")", ":"] → 7 tokens
"I can't" → ["I", " can", "'", "t"] → 4 tokens
Rule of thumb for English: 1 token ≈ 4 characters ≈ 0.75 words
Code is tokenized differently than prose. Indentation, symbols, and keywords each become separate tokens. A 200-line Python file might be 800–1,200 tokens.
2. How Tokenization Works
Most modern LLMs use Byte Pair Encoding (BPE) — a subword tokenization algorithm that learns the most common character patterns from training data.
| Text | Tokens | Count |
|---|---|---|
| "Hello, world!" | Hello | , | world | ! | 4 |
| "https://example.com/path?q=search" | https | : | / | / | example | . | com | / | path | ? | q | = | search | 13 |
| "def calculate_average(numbers):" | def | calculate | _ | average | ( | numbers | ) | : | 8 |
| "price = $19.99" | price | = | $ | 19 | . | 99 | 6 |
Key insight: URLs, code symbols, and punctuation are token-expensive. A URL like https://example.com/path?q=search is 13 tokens — more than the entire sentence "The cat sat on the mat" (7 tokens).
3. Token Counting
Quick estimation rules for English:
• 1 token ≈ 4 characters (including spaces)
• 1 token ≈ 0.75 words
• 1,000 tokens ≈ 750 words ≈ 1 page
• 100,000 tokens ≈ a 300-page book
Code tokens: ~3–4 characters per token (more symbols = more tokens)
| Document Type | Typical Size | Est. Tokens |
|---|---|---|
| Short email | 200 words | ~270 tokens |
| Blog post | 1,500 words | ~2,000 tokens |
| Python file (200 lines) | ~4,000 chars | ~800 tokens |
| Python file (2,000 lines) | ~40,000 chars | ~8,000 tokens |
| Research paper | 8,000 words | ~10,700 tokens |
| Entire novel | 80,000 words | ~107,000 tokens |
4. The Context Window
The context window is the maximum number of tokens an LLM can process in a single request — including BOTH input (your prompt + any retrieved context) AND output (the model's response).
Example: 8K context window
- System prompt: 500 tokens
- User message: 200 tokens
- Retrieved docs: 5,000 tokens
- Available output: 2,300 tokens
If your input uses 7,000 tokens, you can only generate 1,000 output tokens.
What happens when you exceed the context window? The model cannot see tokens beyond the limit. Earlier tokens may be truncated, or the API may return an error. Information is lost — like trying to remember a conversation from last week.
5. Context Budget
Every token in the context window must be allocated. Here is a realistic budget breakdown for a 128K context model:
| Component | Tokens | % of 128K | Purpose |
|---|---|---|---|
| System prompt | 2,000 | 1.6% | Instructions, rules, persona |
| Conversation | 10,000 | 7.8% | Chat history |
| Retrieved context | 80,000 | 62.5% | Documents, code, data |
| Output reserve | 4,096 | 3.2% | Model response space |
| Remaining | 31,904 | 24.9% | Buffer / flexibility |
6. Model Comparison
| Model | Context Window | ~Words | Best For |
|---|---|---|---|
| GPT-3.5 Turbo | 4,096 | ~3K | Short prompts, simple tasks |
| GPT-4 | 8,192 | ~6K | Moderate documents, code review |
| GPT-4 Turbo | 128,000 | ~96K | Long documents, full codebases |
| Claude 3.5 Sonnet | 200,000 | ~150K | Very long documents, analysis |
| Llama 3.1 | 128,000 | ~96K | Local deployment, privacy |
| Gemini 1.5 Pro | 1,000,000+ | ~750K | Entire books, massive datasets |
7. Why More Context Is Not Better
This is the most misunderstood aspect of context windows. Research shows several problems with very long contexts:
The "Lost in the Middle" Problem
If you put the most important information in the middle of a long document, the model may miss it.
Practical Tradeoffs
| More Context ✓ | More Context ✗ |
|---|---|
| More information available | Higher API cost (potentially $1–5+ per call) |
| Better full-document understanding | Slower processing (more tokens = more compute) |
| Fewer truncation issues | "Lost in the middle" — model ignores middle content |
| Better RAG retrieval quality | More hallucination risk with irrelevant context |
| Handles complex, multi-file tasks | Diluted attention — model spreads focus too thin |
8. Tokens in Coding
Code is token-expensive because of symbols, indentation, and keywords:
Single function (50 lines): ~400 tokens → fits in 4K easily
One file (200 lines): ~800 tokens → fits in 4K
Large file (2,000 lines): ~8,000 tokens → needs 32K context
Project (10 files): ~20,000 tokens → needs 32K+ context
Full codebase (50 files): ~80,000 tokens → needs 128K context
AI coding agents (Claude Code, Cursor) manage this automatically.
Practical tip: When asking an AI to review code, provide only the relevant files — not the entire codebase. This reduces cost, improves focus, and avoids the "lost in the middle" problem.
See How AI Coding Agents Actually Work for how agents manage context.
9. Tokens with Documents
Document length directly impacts what models can process:
| Document | ~Tokens | Fits in 8K? | Fits in 128K? |
|---|---|---|---|
| README.md | ~500 | ✅ Yes | ✅ Yes |
| API documentation | ~3,000 | ✅ Yes | ✅ Yes |
| Research paper | ~10,000 | ❌ No | ✅ Yes |
| Legal contract | ~8,000 | ⚠️ Tight | ✅ Yes |
| Book chapter | ~15,000 | ❌ No | ✅ Yes |
| Entire novel | ~100,000 | ❌ No | ⚠️ Tight |
10. Tokens in RAG
RAG (Retrieval-Augmented Generation) solves the context window problem by retrieving only relevant chunks instead of loading entire documents.
User query → Load entire 100K document → 95K tokens wasted on irrelevant content
WITH RAG:
User query → Embed query → Find top 5 relevant chunks → Load only 5K tokens
→ Model sees exactly what it needs → Better answer, lower cost
RAG is the standard approach when your knowledge base exceeds the context window. It retrieves the most relevant chunks and sends only those to the model.
Learn more in RAG Architecture Explained.
11. Tokens in AI Agents
AI coding agents face unique token challenges because they manage multiple tool calls, file reads, and code edits in a single session:
| Agent Action | Typical Tokens | Impact |
|---|---|---|
| Read a file | 500–5,000 | Consumes input budget |
| Run a command | 100–1,000 | Output tokens count |
| Edit a file | 200–2,000 | Both input and output |
| Run tests | 500–3,000 | Test output is token-heavy |
| Full agent session | 50,000–200,000 | Accumulates across turns |
Agents must strategically manage context — reading only necessary files, summarizing old outputs, and avoiding unnecessary back-and-forth.
See Context Engineering Explained for how developers control what the model sees.
12. Context Management Strategies
| Strategy | How It Works | Best For |
|---|---|---|
| Truncation | Keep first N tokens | Simple, but loses end context |
| Sliding Window | Keep recent N tokens | Long conversations |
| RAG | Retrieve relevant chunks | Large knowledge bases |
| Summarization | Summarize old context | Reducing conversation history |
| Chunking | Split into pieces, process separately | Parallel processing |
| Hierarchical | Summary + on-demand details | Overview + drill-down |
13. Cost Implications
Tokens directly determine API costs. Here are illustrative costs per 1M tokens:
| Scenario | Input Tokens | Output Tokens | GPT-4 Cost |
|---|---|---|---|
| Short prompt | 500 | 200 | ~$0.01 |
| Document analysis | 10,000 | 2,000 | ~$0.16 |
| Long document | 100,000 | 5,000 | ~$1.15 |
| Full codebase | 500,000 | 10,000 | ~$5.30 |
Key insight: Using RAG to send 5K relevant tokens instead of 100K full-document tokens can reduce cost by 95% while improving answer quality.
14. FAQ
Are tokens the same as words?
How do I count tokens before sending a request?
What happens if I exceed the context window?
Why is more context not always better?
What is the "lost in the middle" problem?
Should I always use the largest context window available?
How does RAG help with context limits?
Try These BestWordz Tools
- Regex Tester — Practice pattern matching, useful for understanding tokenization rules
- JSON Formatter — Format API responses that include token counts
- All BestWordz Tools — Explore the complete tool library
Continue Learning
- What Is an LLM? Beginner's Guide — Foundational concepts
- How LLMs Generate Text — Tokens, probabilities, and generation
- Transformers Explained — The architecture behind context processing
- Embeddings Explained — How tokens become vectors
- Context Engineering Explained — Controlling what the model sees
- Prompt Engineering Tutorial — Writing efficient prompts
- RAG Architecture Explained — Retrieval-augmented generation
- How AI Coding Agents Work — Token management in agents
Try the JSON Formatter
Put what you've learned into practice with this free BestWordz tool.
💬 Discuss this topic
Have questions or insights about What Are Tokens?? Join the BestWordz Community.
Continue Learning: Prompt Engineering
Master the art of communicating with AI
- Free-Form vs Structured Output
- What Are Tokens? (this article)
- The Sequence Modeling Problem
- AI → Machine Learning → Deep Learning
- The 10-Stage CS Learning Roadmap
📚 Related Articles
AI → Machine Learning → Deep Learning
Key Takeaway A Large Language Model (LLM) is a neural network trained on massive text data to predi…
AI & Machine LearningThe Sequence Modeling Problem
Key Takeaway Transformers process all tokens simultaneously using self-attention — a mechanism that…
CybersecurityFrom Prompt Crafting to System Design
Key Takeaway --> 🎯 Context engineering is the skill of designing what an AI system knows, s…
CybersecurityThe Problem: AI Without Context
Key Takeaway --> 🎯 RAG retrieves relevant knowledge from your documents. MCP connects AI ag…
CybersecurityThe 10-Stage CS Learning Roadmap
A computer science education in 2026 requires more than traditional coursework. Today's students ne…
AI & Machine LearningThe Complete Pipeline
Key Takeaway LLMs generate text one token at a time through a repeating cycle: tokenize input → com…
🔧 Related Tools
JSON Formatter
Pretty-print or minify any JSON document instantly, with clear line/column error reporting.
Try it now →Regex Tester
Test regular expressions live: matches with positions, capture groups, and flag validation.
Try it now →Certificate Decoder
Decode and parse X.509 certificates with structured output.
Try it now →Diffie-Hellman Demo
Educational demonstration of classic Diffie-Hellman key exchange.
Try it now →💬 Discuss on BestWordz Community
Join the conversation about Python, LLMs, GPT on the BestWordz Community forum.
Visit Forum →