What is tokenisation?
Tokenisation is the process by which text is broken down into the smallest processing units (tokens). Language models do not think in words but in tokens, which roughly correspond to 3–4 characters.
DEFINITION
Language models do not read words the way humans do. They first break text down into tokens: small units of letters, syllables or common word parts. “Manager” is split into several tokens, for example, while “the” is a single token. One token corresponds roughly to 0.75 English words or 3–4 characters.
Why does this matter? Because every language model has a context window: a maximum number of tokens it can process at once. Texts that are too long are truncated or cause errors. Costs for API use are calculated per token. And the efficiency of prompts depends on how much useful information each token carries.
For users: long, redundant prompts cost more and often achieve less than precise, dense formulations.
CONNECTIONS
Leadership
Anyone introducing AI-supported communication should understand why very long inputs can lead to worse results. Tokenisation explains why precision in wording matters more than verbosity.
Agility
When using AI for user stories and backlog work, the context window influences how many entries can be analysed at once. Backlog blocks that are too large must be processed in parts.
Project Management
In projects with long documentation (charters, contracts, reports), the context window determines how much an AI assistant can keep in view at once. Large documents need RAG systems rather than simple prompts.
KEY POINTS
- Language models process tokens, not words or sentences.
- One token ≈ 0.75 English words ≈ 3–4 characters.
- The context window limits how much text a model can process at once.
- API costs are calculated by the number of tokens processed.
- Precise prompts are more efficient than long, redundant text.
EXAMPLE
A project manager wants an AI to summarise a 50-page project report. They paste the report directly into the chat. The system returns an error: “Context length exceeded”. They have exceeded the context window. Solution: split the report into chapters, summarise each separately and then consolidate the summaries. Or: use a RAG system that addresses relevant sections selectively.
MISCONCEPTIONS
Is a longer context window always better?
Not necessarily. Very long contexts can lead to “lost in the middle”: the model loses relevant information from the middle of long inputs. Selectively including relevant sections is often better than submitting complete documents.
Are tokens the same in all models?
No. Different models differ significantly in tokenizer design and context window size. GPT-4 and Claude tokenise the same text differently.