Understanding Tokens

If you've used AI tools, you've probably heard about "tokens." But what exactly are they? This guide explains tokens in depth, with interactive tools to help you understand how your text gets tokenized.

Try It Yourself

Use this tool to see exactly how your text gets broken into tokens:

What Is a Token?

A token is the basic unit that AI models use to process text. When you send a message to an AI, it doesn't read characters or words—it reads tokens.

Think of tokens as the "vocabulary" the AI uses. Just like how humans break down sentences into words, AI models break down text into tokens. But tokens don't always match our intuition about words.

Tokens Are Not Words

Here's where it gets interesting:

Notice how:

• Common words are often 1 token
• Capitalization can change tokenization
• Long or uncommon words get split into pieces
• Technical terms may become multiple tokens

The 4-Character Rule (and Why It's Wrong)

You'll often hear "1 token ≈ 4 characters" as a quick estimate. This is a useful approximation, but it's not accurate:

The approximation works reasonably well for plain English text, but breaks down with:

• Emojis: Each emoji can be 2-3 tokens
• Non-English languages: Often more tokens per character
• Code: Varies widely based on syntax and naming
• Numbers: Can be tokenized digit-by-digit

Input vs Output Tokens

When you interact with AI, there are two types of tokens:

Input Tokens (Prompt Tokens)

Everything you send to the AI:

• Your message
• System instructions (set by the app builder)
• Conversation history
• Any context or documents

Output Tokens (Completion Tokens)

Everything the AI generates:

• The response text
• Any formatted content (markdown, code, etc.)

Why This Matters for Pricing

Output tokens typically cost 2-4x more than input tokens. Why?

1. Input: The model reads your tokens (relatively fast)
2. Output: The model generates tokens one-by-one (computationally intensive)

This is why shorter, more concise AI responses cost less than long, verbose ones.

How Tokenization Works

AI models use a technique called Byte Pair Encoding (BPE) to create their vocabulary. Here's a simplified explanation:

1. Start with characters: Begin with individual characters as the base vocabulary
2. Find common pairs: Look for character pairs that appear frequently together
3. Merge pairs: Combine the most common pairs into single tokens
4. Repeat: Continue merging until you have a vocabulary of ~100,000 tokens

The result is a vocabulary where:

• Very common words ("the", "is", "and") are single tokens
• Common word parts ("-ing", "-tion", "un-") are single tokens
• Rare words get split into known pieces

Different Models, Different Tokenizers

Not all AI models tokenize text the same way:

This means the same text may have different token counts across models, though the differences are usually small for English text.

Practical Tips for Managing Tokens

1. Be Concise in Your Instructions

Shorter system prompts = fewer input tokens on every message.

Verbose (45 tokens):

"I would like you to act as a helpful assistant that can answer questions about various topics. Please provide detailed and informative responses to any questions that users might ask you."

Concise (15 tokens):

"You're a helpful assistant. Provide detailed, informative answers."

2. Consider Response Length

If you don't need long responses, tell the AI:

• "Answer in 2-3 sentences"
• "Be brief"
• "Summarize in one paragraph"

3. Watch Conversation Length

Every message in your conversation history counts as input tokens. Longer conversations = more tokens.

4. Code and Technical Content

Code often tokenizes efficiently because:

• Common keywords (function, return, if) are single tokens
• But variable names and strings add up quickly

Token Limits (Context Windows)

Every AI model has a maximum number of tokens it can process at once—this is called the context window.

The context window must fit:

• System prompt
• Conversation history
• Your current message
• Space for the response

If you exceed the limit, older messages get truncated from the conversation.

Cost Examples

Let's look at real token costs using GPT-5 ($6.50/M input,$26/M output):

Example 1: Quick Q&A

• Your question: 50 tokens ($0.000325)
• AI response: 100 tokens ($0.0026)
• Total: $0.003 (⅓ of a cent)

Example 2: Long Document Analysis

• Document + question: 5,000 tokens ($0.0325)
• Detailed response: 1,000 tokens ($0.026)
• Total: $0.06 (6 cents)

Example 3: Extended Conversation

• 20 back-and-forth messages
• Average input per turn: 2,000 tokens (includes history)
• Average output: 300 tokens
• Total: ~$0.40 (40 cents)