Tokens

Tokens are the fundamental units that language models use to process and generate text. Rather than working with individual characters or whole words, LLMs break text into tokens — which are typically words, common word parts, or individual characters for rare words.

How tokenization works: "Hello, how are you?" becomes roughly 5 tokens: ["Hello", ",", " how", " are", " you", "?"]. "Unbelievable" might become 3 tokens: ["Un", "believ", "able"]. Numbers, punctuation, and special characters each consume tokens.

Key token facts: 1 token is approximately 4 characters or 0.75 words in English, 100 tokens is approximately 75 words, a typical page of text is roughly 300-400 tokens, other languages often require more tokens per word (Chinese, Japanese, Korean), and code tends to be token-heavy due to syntax characters.

Tokens matter for AI agent builders because: pricing is per token (input tokens + output tokens), context windows have token limits (not word limits), response length is measured in tokens, and knowledge base chunks are sized by tokens.

Token economics: each conversation turn consumes tokens for the system prompt, conversation history, retrieved knowledge, user message, and AI response. A typical AI agent conversation of 10 turns might consume 5,000-20,000 tokens, depending on system prompt length, knowledge retrieval, and response verbosity.

Builders can manage token costs by: keeping system prompts concise, retrieving relevant (not excessive) knowledge, summarizing long conversation histories, and choosing appropriate model sizes for different tasks.