A token is the smallest unit of text a language model handles. Models do not read letters and they do not read words. They read integers pulled from a fixed vocabulary, and each integer is a token. Text entering a model is split into that sequence; text leaving it arrives as integers reassembled into characters. The unit exists for a mechanical reason. A neural network needs an input alphabet that is finite and fixed before training begins, while written language is open-ended, full of names, typos, code and scripts the training data never contained. Tokens sit between characters and words, small enough to cover anything and large enough to keep sequences short.

the PPC Land daily grid

Think you know ad tech? Prove it. PPC Land now runs a daily word game built entirely from the language of programmatic - sixteen terms, four hidden groups of four, one fresh grid every morning. Some tiles look like they belong somewhere they don't, and that misdirection is the whole puzzle. There's a weekly crossword too, drawn from the terminology that fills briefs, DSP dashboards, and measurement decks. Free to play, no account needed. Find out whether you really know your bid shading from your supply path optimization.

Learn more

The unit also became a currency. Every commercial model API meters usage by the token, so the number that governs what a model can hold in memory also governs what a creative pipeline or an agentic buying stack costs to run.

How text becomes tokens

Most models in production use byte pair encoding, usually shortened to BPE. Training a tokenizer starts from raw bytes and repeatedly merges the most frequent adjacent pair into a new symbol, stopping when the vocabulary reaches a target size. Frequent strings end up as single tokens, rare ones stay split into pieces.

According to the documentation for OpenAI's tiktoken library, the encoding is reversible and lossless, works on text outside the tokenizer's training data, compresses, and averages roughly four bytes per token. OpenAI's own guidance puts one token at about four characters, or three-quarters of a word.

Vocabulary size sets the compression rate. GPT-2 shipped in February 2019 with 50,257 entries. The cl100k_base encoding used by the GPT-4 generation holds about 100,000, and its successor o200k_base, introduced in 2024, about 200,000, which shortens sequences noticeably for text outside English.

Splits are not intuitive. Capitalisation and leading whitespace change the result, so "red", "Red" and " red" can each map to different tokens, and numbers fragment unpredictably. A 1,500-word article lands near 2,000 tokens; a 100,000-token document is roughly a short book.

Where tokens get counted

Providers meter several distinct streams. Input tokens cover everything sent to the model: system instructions, tool definitions, retrieved documents, conversation history. Output tokens cover what comes back, and reasoning models add internal deliberation billed at the output rate but often never shown to the caller. Cached input tokens cover prefixes the provider has already processed and stored.

Prices separate sharply between streams. Anthropic's published rates as of August 2026 put Claude Opus 5 at $5 per million input tokens and $25 per million output, Claude Sonnet 5 at $2 and $10, and Claude Haiku 4.5 at $1 and $5. Google's Gemini 3.6 Flash launched on July 21, 2026 at $1.50 input and $7.50 output, with Gemini 3.5 Flash-Lite at $0.30 and $2.50. Output runs at five times input across all of them, which is why verbose models cost more than their headline rate suggests. A million output tokens is around 750,000 words, six or seven full-length novels.

Caching changes the arithmetic again. Anthropic charges 1.25 times the base input rate to write a five-minute cache, twice the base rate for a one-hour cache, and one-tenth to read from either, with minimum cacheable prefixes running from 512 to 4,096 tokens depending on the model.

Two other limits use the same unit. The context window is the maximum number of tokens a model can attend to at once, and rate limits are published as tokens per minute.

Origin and evolution

Byte pair encoding was not invented for language models. Philip Gage published it as a data compression algorithm in the February 1994 issue of The C Users Journal. Google researchers Mike Schuster and Kaisuke Nakajima described a related subword method, WordPiece, in a 2012 paper on Japanese and Korean voice search.

The adaptation that matters for current systems came from Rico Sennrich, Barry Haddow and Alexandra Birch, who posted "Neural Machine Translation of Rare Words with Subword Units" to arXiv in August 2015 and presented it at ACL in Berlin in August 2016. Translation systems until then carried fixed word vocabularies and failed on anything outside them. Subword units made open-vocabulary translation possible. Taku Kudo and John Richardson released SentencePiece in 2018, removing the assumption that text arrives pre-split on whitespace and making the approach workable for Japanese and Chinese. GPT-2 moved the operation down to raw bytes in 2019, guaranteeing that any input encodes.

Commercial metering followed the technique. The GPT-3 API, opened in June 2020, priced access per token, and every major provider since has copied the structure. Prompt caching arrived across the main APIs in 2024, with Anthropic claiming cost reductions of up to 90% on long prompts. By 2026 the competitive claim had shifted from price per token to tokens consumed: Google says Gemini 3.6 Flash produces 17% fewer output tokens than its predecessor on the Artificial Analysis Index, and up to 65% fewer on the DeepSWE benchmark.

Why the unit reaches media buying

Agency finance is where the counting became visible. Omnicom's third-party service costs reached $2.9 billion in the first half of 2026 against $1.7 billion a year earlier, a 71% rise, and chief executive John Wren said on July 29, 2026 that "the marketplace hasn't seen what the cost of this AI is". Ebiquity chief executive Ruben Schreurs was blunter, saying agencies "can't afford to continue to subsidize all those costs". Some pitches now bundle free AI tokenisation into principal media commitments, while a single demonstration task across three agents at an IAB Europe showcase in April 2026 ran up around 50 euros in token charges.

Autonomous software is why the meter spins faster. Data compiled from OpenRouter and circulated in August 2026 shows agents consuming close to five times the tokens per task that human users do, with agent usage up roughly fourteenfold since February 2026 and more than 85% of that consumption coming from cached prompts, since agents reload tool definitions, system prompts and schema constraints on every turn. Draft Digital's Lars Postmus, describing a buying stack that needed 12 separate Model Context Protocol calls across five roles, wrote that "tokens burn on every call, and the strategy quietly fades on the way down".

Cost visibility separates the projects that survive from the ones that stall. A KPMG survey of 2,145 senior leaders conducted between April 28 and May 25, 2026 found 49% had scaled back, narrowed, delayed or paused agent deployments. Only 35% reported full visibility into AI operating costs, and among those 15% had established return on investment against 3% of the rest. Magnite chief executive Michael Barrett has capped his 2027 expectation for agentic ad spend at $600 million to $700 million.

Limitations and disputes

Token counts are not comparable across vendors, which makes per-token prices misleading on their own. One Tamil document in the ONERULER benchmark measured 42,124 tokens under Gemini's tokenizer and 103,990 under Qwen's. Anthropic's documentation notes that Claude 4.7 and later models produce roughly 30% more tokens for the same text, so a headline price cut can be cancelled by a tokenizer change.

The disparity falls unevenly by language. Aleksandar Petrov, Emanuele La Malfa, Philip Torr and Adel Bibi found tokenization lengths for the same text translated into different languages differing by up to 15 times, in work presented at NeurIPS in 2023. Because cost, latency and usable context all scale with token count, speakers of some languages pay materially more for identical work. Long context widens the gap: the ONERULER researchers measured an 11% accuracy spread across languages at 8,000 tokens growing to 34% at 128,000.

Billing itself has proved fragile. A Google bug confirmed on August 25, 2025 miscategorised internal multimodal thinking tokens as high-cost image output, with one developer reporting more than $70,000 in erroneous charges. Google added project spend caps to the Gemini API on March 16, 2026, carrying a delay of roughly ten minutes during which the customer remains liable for overages.

A deeper objection targets the layer itself. Because a model never sees the characters inside a token, tokenization is why systems miscount letters, stumble on arithmetic and handle rhyme inconsistently. Byte-level and tokenizer-free architectures remain an active research line, though no production system at comparable scale has dispensed with the layer.

Disambiguation

Private State Tokens, formerly Trust Tokens, are cryptographic anti-fraud signals in Chrome's Privacy Sandbox. A browser stores up to 500 per issuer and site, and a redeemer may spend two per device and issuer every 48 hours. They carry trust, not text.

UID2 tokens are the encrypted form of a raw Unified ID 2.0 identifier, generated by an operator from a hashed email or phone number and sent in the bidstream with a recommended hourly refresh. The encryption is the point; nothing is counted.

Access tokens are authorisation credentials in the OAuth sense. IAB Tech Lab's Content Monetization Protocols specification, released on March 10, 2026, issues one to permit content ingestion. It grants permission rather than metering volume.

Blockchain tokens are transferable units of value on a distributed ledger, a usage that reached advertising through several attempts at cryptographic media exchanges and shares only the word.

Recent developments

Price movement has slowed at the top of the market while consumption keeps climbing. Anthropic's documentation states that the introductory $2 and $10 rates for Claude Sonnet 5 became standard rather than rising to $3 and $15 in September 2026, and Google cut Gemini Flash pricing in July 2026 while leaning on output efficiency rather than headline rate. The August 2026 analysis makes the same point from the demand side: token pricing stopped declining while usage accelerated, leaving cached prompts as the mechanism holding agent economics together. Agencies, roughly two years after absorbing model costs as an acquisition expense, have started treating the unit as a revenue line.

Timeline

  • February 1994: Philip Gage publishes byte pair encoding as a compression algorithm in The C Users Journal
  • 2012: Mike Schuster and Kaisuke Nakajima describe WordPiece subword segmentation for Japanese and Korean voice search
  • August 2015: Rico Sennrich, Barry Haddow and Alexandra Birch post "Neural Machine Translation of Rare Words with Subword Units" to arXiv
  • August 2016: The subword paper is presented at ACL in Berlin
  • 2018: Taku Kudo and John Richardson release SentencePiece, removing the whitespace pre-tokenization requirement
  • February 2019: GPT-2 ships with byte-level BPE and a 50,257-token vocabulary
  • June 2020: The GPT-3 API opens with per-token pricing
  • 2023: Aleksandar Petrov and co-authors present evidence of up to 15-fold tokenization differences between languages at NeurIPS
  • 2024: Prompt caching reaches the major model APIs; the o200k_base encoding roughly doubles vocabulary size
  • March 3, 2025: The ONERULER multilingual long-context benchmark is announced
  • August 25, 2025: Google confirms a billing bug miscategorising thinking tokens as image output
  • March 16, 2026: Google adds project spend caps to the Gemini API
  • July 21, 2026: Gemini 3.6 Flash launches at $1.50 input and $7.50 output per million tokens
  • August 2026: OpenRouter data shows agents consuming close to five times the tokens per task that humans do

Summary

Who. Model providers set token prices and publish the tokenizers that determine counts. Advertisers, agencies, ad tech vendors and publishers pay those prices through every AI feature, copilot and autonomous buying agent they operate. Researchers at Edinburgh, Oxford, Google, Maryland and elsewhere built and continue to critique the underlying encoding.

What. A token is a subword unit of text produced by an encoding scheme, most often byte pair encoding. It is simultaneously the input format of a language model, the limit on how much a model can consider at once, and the billing unit of the model economy. One token averages about four characters of English.

When. The algorithm dates to 1994 as compression and to 2015 and 2016 as a language method. Per-token commercial pricing began in June 2020. The unit became a line item in advertising budgets during 2025 and 2026, as agentic systems moved from demonstration into media buying workflows.

Where. Tokens are counted inside model APIs and passed through every layer built on them: agency AI platforms, demand-side platform copilots, creative generation tools, publisher answer engines and the Model Context Protocol integrations connecting them.

Why. The unit matters because it converts computational work into money at a rate few buyers can audit. Counts differ across vendors and languages, output costs several times input, reasoning is billed without being shown, and agentic architectures multiply consumption on every turn. Marketing organisations that cannot measure token use cannot forecast the cost of the automation they are buying.