Understanding Tokens

If you’ve used AI tools, you’ve probably heard about “tokens.” But what exactly are they? This guide explains tokens in depth, with interactive tools to help you understand how your text gets tokenized.

Try It Yourself

Use this tool to see exactly how your text gets broken into tokens:

What Is a Token?

A token is the basic unit that AI models use to process text. When you send a message to an AI, it doesn’t read characters or words: it reads tokens.

Think of tokens as the “vocabulary” the AI uses. Just like how humans break down sentences into words, AI models break down text into tokens. But tokens don’t always match our intuition about words.

Tokens Are Not Words

Here’s where it gets interesting:

Text	Tokens	Count
”hello”	hello	1
”Hello”	Hello	1
”HELLO”	HE, LLO	2
”tokenization”	token, ization	2
”ChatGPT”	Chat, G, PT	3

Notice how:

• Common words are often 1 token
• Capitalization can change tokenization
• Long or uncommon words get split into pieces
• Technical terms may become multiple tokens

The 4-Character Rule (and Why It’s Wrong)

You’ll often hear “1 token ≈ 4 characters” as a quick estimate. This is a useful approximation, but it’s not accurate:

Text	Characters	Actual Tokens	4-Char Estimate
”The quick brown fox”	19	4	5
”supercalifragilisticexpialidocious”	34	9	9
”🎉🎊🎁“	3	6	1
”日本語”	3	3	1
console.log("hello")	21	7	5

The approximation works reasonably well for plain English text, but breaks down with:

• Emojis: Each emoji can be 2-3 tokens
• Non-English languages: Often more tokens per character
• Code: Varies widely based on syntax and naming
• Numbers: Can be tokenized digit-by-digit

Input vs Output Tokens

When you interact with AI, there are two types of tokens:

Input Tokens (Prompt Tokens)

Everything you send to the AI:

• Your message
• System instructions (set by the app builder)
• Conversation history
• Any context or documents

Output Tokens (Completion Tokens)

Everything the AI generates:

• The response text
• Any formatted content (markdown, code, etc.)

Why This Matters for Pricing

Output tokens typically cost 2-4x more than input tokens. Why?

1. Input: The model reads your tokens (relatively fast)
2. Output: The model generates tokens one-by-one (computationally intensive)

This is why shorter, more concise AI responses cost less than long, verbose ones.

How Tokenization Works

AI models use a technique called Byte Pair Encoding (BPE) to create their vocabulary. Here’s a simplified explanation:

1. Start with characters: Begin with individual characters as the base vocabulary
2. Find common pairs: Look for character pairs that appear frequently together
3. Merge pairs: Combine the most common pairs into single tokens
4. Repeat: Continue merging until you have a vocabulary of ~100,000 tokens

The result is a vocabulary where:

• Very common words (“the”, “is”, “and”) are single tokens
• Common word parts (“-ing”, “-tion”, “un-”) are single tokens
• Rare words get split into known pieces

Different Models, Different Tokenizers

Not all AI models tokenize text the same way:

Model Family	Tokenizer	Vocabulary Size
GPT-4.1, GPT-5.x	o200k_base	~200,000
Claude 4.x	Claude tokenizer	~100,000
Gemini 3.x	SentencePiece	~256,000

This means the same text may have different token counts across models, though the differences are usually small for English text.

Practical Tips for Managing Tokens

1. Be Concise in Your Instructions

Shorter system prompts = fewer input tokens on every message.

Verbose (45 tokens):

“I would like you to act as a helpful assistant that can answer questions about various topics. Please provide detailed and informative responses to any questions that users might ask you.”

Concise (15 tokens):

“You’re a helpful assistant. Provide detailed, informative answers.”

2. Consider Response Length

If you don’t need long responses, tell the AI:

• “Answer in 2-3 sentences”
• “Be brief”
• “Summarize in one paragraph”

3. Watch Conversation Length

Every message in your conversation history counts as input tokens. Longer conversations = more tokens.

4. Code and Technical Content

Code often tokenizes efficiently because:

• Common keywords (function, return, if) are single tokens
• But variable names and strings add up quickly

Token Limits (Context Windows)

Every AI model has a maximum number of tokens it can process at once (the context window). Models also have a max output limit on how many tokens they can generate in a single response.

Model	Provider	Context window	Max output
GPT-5.4	OpenAI	128,000 tokens	32,768 tokens
GPT-4.1	OpenAI	1,047,576 tokens	32,768 tokens
GPT-4.1 Nano	OpenAI	1,047,576 tokens	32,768 tokens
Claude Opus 4.8	Anthropic	1,000,000 tokens	128,000 tokens
Claude Sonnet 4.6	Anthropic	1,000,000 tokens	64,000 tokens
Claude Haiku 4.5	Anthropic	200,000 tokens	64,000 tokens
Gemini 3.1 Pro Preview	Google	1,048,576 tokens	65,536 tokens
Gemini 3.5 Flash	Google	1,048,576 tokens	65,536 tokens
o3	OpenAI	200,000 tokens	65,536 tokens
o4 Mini	OpenAI	200,000 tokens	65,536 tokens

The context window must fit:

• System prompt
• Conversation history
• Your current message
• Space for the response

If you exceed the limit, older messages get truncated from the conversation.

Model Pricing Comparison

Prices are per 1 million tokens, including the Chipp platform markup:

Model	Provider	Input per 1M tokens	Output per 1M tokens
GPT-5.4	OpenAI	$3.25	$19.50
GPT-4.1	OpenAI	$2.60	$10.40
GPT-4.1 Nano	OpenAI	$0.13	$0.52
Claude Opus 4.8	Anthropic	$6.50	$32.50
Claude Sonnet 4.6	Anthropic	$3.90	$19.50
Claude Haiku 4.5	Anthropic	$1.30	$6.50
Gemini 3.1 Pro Preview	Google	$2.60	$15.60
Gemini 3.5 Flash	Google	$1.95	$11.70
o3	OpenAI	$2.60	$10.40
o4 Mini	OpenAI	$1.43	$5.72

Cost Examples

Here is what a month of traffic costs on GPT-5.4, at today’s prices. Cost = input tokens x input price + output tokens x output price.

Example 1: Quick Q&A

A 50-token question and a 100-token answer, 1,000 times a month:

Estimated cost on GPT-5.4: about $2.11 per month for 1,000 messages of 50 input and 100 output tokens each ($3.25 input and $19.50 output per 1M tokens).

Example 2: Long Document Analysis

A 5,000-token document and question with a 1,000-token answer, 100 times a month:

Estimated cost on GPT-5.4: about $3.58 per month for 100 messages of 5,000 input and 1,000 output tokens each ($3.25 input and $19.50 output per 1M tokens).

Example 3: Extended Conversation

A 20-turn conversation averages about 2,000 input tokens per turn (the history grows) and 300 output tokens. For 100 conversations a month (2,000 turns):

Estimated cost on GPT-5.4: about $24.70 per month for 2,000 messages of 2,000 input and 300 output tokens each ($3.25 input and $19.50 output per 1M tokens).

Budget-Friendly Alternative (GPT-4.1 Nano)

The same conversations on GPT-4.1 Nano:

Estimated cost on GPT-4.1 Nano: about $0.83 per month for 2,000 messages of 2,000 input and 300 output tokens each ($0.13 input and $0.52 output per 1M tokens).