Token

A token is the atomic unit that large language models work with. When you send text to an AI model, it is first broken into tokens through tokenization. The model processes these tokens, performs its computations, and generates new tokens as output. Understanding tokens is essential for working effectively with AI APIs and for understanding the costs and constraints of language models.

A rough rule of thumb for English text is that one token is approximately 0.75 words, or about 4 characters. The sentence 'The quick brown fox' would typically become 4-5 tokens. Common words like 'the,' 'is,' and 'of' are usually single tokens. Rarer words, technical terms, and words in non-English languages are often split into multiple tokens. For example, 'tokenization' might be split into 'token' and 'ization.'

Tokens are central to both the economics and the capabilities of language model APIs. Most providers charge per token for both input and output. A typical API call might use 500 input tokens (your prompt and context) and generate 300 output tokens (the model's response). Understanding token usage helps you optimize prompts to be effective while managing costs at scale.

The context window of a language model is measured in tokens. A model with a 128,000-token context window can process up to roughly 96,000 words of input at once - about the length of a full novel. Context window size determines how much text the model can 'see' at any time, affecting its ability to maintain coherence over long conversations or analyze lengthy documents.

Token efficiency is an important consideration when building AI-powered applications. Verbose prompts, excessive repetition of context, and long conversation histories all consume tokens unnecessarily. Techniques like prompt compression, caching repeated context, and summarizing long conversation histories help reduce token usage while maintaining model performance - important for cost management in production AI systems.