Inference cost is what it costs to run a trained artificial intelligence (AI) model every time someone uses it. Each answer from a chatbot, each AI-generated search summary and each agent that books media consumes processor time, memory, electricity and data centre capacity. That bill is the inference cost. The term exists because it behaves differently from the cost of building a model. Inference itself is the act of running the model; its cost is the recurring expense that follows a model into production and grows with every user it gains.

Where the money goes

The bill starts with hardware. Large language models (LLMs) run on graphics processing units (GPUs) or custom accelerators, rented by the hour or owned and depreciated over several years. Cost per request depends on how many requests each chip handles at once, how much of its capacity is used, the price of electricity and how much spare capacity sits idle for traffic spikes. An accelerator running at a third of capacity makes every answer roughly three times as expensive as it needs to be.

Google offered a rare look at the full stack in August 2025. In a post by Amin Vahdat and Jeff Dean, it said the median Gemini app text prompt used 0.24 watt-hours once idle machines, host processors and cooling were counted, against 0.10 watt-hours for the active accelerators alone, a figure Google called "an optimistic scenario at best." Much of the cost of an answer sits outside the model's own arithmetic. Google also reported a 33-fold fall in energy per median prompt between May 2024 and May 2025, unverified by any third party.

Length adds a second layer. Models read and write in tokens, the subword units they process and bill by. Reading a prompt happens in parallel and is comparatively cheap. Writing happens one token at a time, so output costs more, and price lists reflect it. Techniques such as context caching, quantisation and mixture-of-experts designs exist mainly to cut one side of this ledger or the other.

How the cost is priced

Most buyers never see a GPU bill. They see a per-token price, quoted in US dollars per million tokens and split between input and output. Anthropic released Haiku 5.5 on October 7, 2026 at $0.10 per million input tokens and $0.50 per million output tokens for prompts up to 100,000 tokens, 90% below Haiku 4.5, with longer prompts billed at five times those rates. Google priced Gemini 4 Argon at $2 input and $10 output on October 13, 2026, calling the rate introductory.

A worked example shows how those prices turn into cost per query. A request with a 2,000-token prompt and a 500-token answer costs $0.00045 at Haiku 5.5 rates, or 45 cents per thousand queries. The same request on Gemini 4 Argon costs about $0.009, close to a cent. Add 10,000 tokens of hidden reasoning and the Argon request rises to roughly 11 cents. That last case is not hypothetical. According to OpenAI's developer documentation, reasoning models generate anywhere from a few hundred to tens of thousands of internal reasoning tokens per response; they are invisible to the user but billed as output tokens. Google has started marketing token thrift as a feature, saying Gemini 3.6 Flash uses 17% fewer output tokens than its predecessor.

List prices are not the same as cost. A provider can price below its serving cost to win share, or above it to recover training spend, and flat subscriptions cover wildly different levels of use. Sam Altman, chief executive of OpenAI, wrote in January 2025 that the company was "currently losing money on OpenAI Pro subscriptions," its $200-a-month tier, because subscribers used it more than expected. Google moved the other way on October 9, 2026, when it cut free Gemini accounts to one model, Flash-Lite, leaving the more expensive Flash and Pro models to paying tiers.

Origin and evolution

Machine learning always had serving costs, but the term entered business vocabulary with ChatGPT, released on November 30, 2022. Within days Altman called its computing costs "eye-watering." In February 2023, Reuters reported that an exchange with a language model likely cost around 10 times a standard keyword search, according to John Hennessy, chairman of Alphabet. Morgan Stanley put a 2022 Google search at roughly a fifth of a cent, and analysts estimated that chatbot-style answers on half of Google's queries could add $6 billion of expense by 2024. Hennessy called the problem "a couple year problem at worst."

Prices then fell faster than almost anyone forecast. Guido Appenzeller of Andreessen Horowitz wrote in November 2024 that the price of a model of equivalent capability was falling roughly 10 times a year, citing a drop from $60 per million tokens for GPT-3 in November 2021 to $0.06 for a comparable small model three years later. Altman made the same 10-times-a-year claim in his "Three Observations" post in February 2025. Epoch AI, a research group, was more cautious on March 12, 2025: depending on the benchmark, prices fell between 9 and 900 times a year, with the fastest drops recent and least certain to persist.

Cheaper tokens did not mean cheaper products. Reasoning models, introduced from late 2024, spend far more tokens per answer, and agents chain dozens of calls together. A Paligo investigation reported by PPC Land in April 2026 described inference costs doubling every quarter, with the ongoing cost of serving deployed models exceeding what it took to build them.

Who pays

Model developers and cloud providers carry the hardware. Sundar Pichai, Alphabet's chief executive, said in April 2026 that 2026 capital expenditure would fall between $175 billion and $185 billion and that Google was "supply-constrained." Platforms building AI into products pay through infrastructure contracts: Pinterest committed $4 billion to Amazon Web Services through 2031, moving feed inference to Amazon's Graviton chips to lower per-query cost.

Agencies now meet inference cost as a line item. PMG sets daily token caps per user and Brainlabs runs a tiered allowance, as PPC Land reported when agencies began metering AI agents by the token. Holding companies have folded token costs into principal media arrangements. Ruben Schreurs, chief executive of Ebiquity, said "agencies can't afford to continue to subsidize all those costs."

Why it matters for advertising

Inference cost is the reason advertising has arrived inside chatbots. A search engine serving a page of links spends a fraction of a cent per query and earns from ads alongside it. A chatbot spends more per answer, and most of its users pay nothing. OpenAI was projected to lose about $14 billion in 2026, according to Reuters, and told investors to expect about $2.5 billion of ad revenue that year. ChatGPT ads launched in the US in February 2026 at a $60 cost per thousand impressions (CPM), and CPMs had fallen to $25 by April. By August, OpenAI said the business had passed a $1 billion annualised run rate.

The arithmetic is tight. At a $25 CPM, one impression earns 2.5 cents. Similarweb recorded sponsored placements in 26% of responses shown to free and Go users, so the average answer earns well under a cent from ads, close to the list cost of a mid-priced model answering a short query and far below a reasoning-heavy one. That squeeze shapes ad load and pushes free users towards cheaper models. Anthropic took the opposite position, pledging on February 4, 2026 to keep Claude ad-free, which leaves subscriptions and API fees to carry its inference bill. The freemium model, with ads or without, is now largely a question of who covers the cost of free answers.

Limitations and disputes

The central dispute is whether inference is actually getting cheaper. Cory Doctorow argued in September 2026 that it is not, and that heavier use deepens losses, citing leaked OpenAI financials reported by Ed Zitron. Epoch AI and Andreessen Horowitz measure the opposite: steep falls in the price of a fixed level of capability. Both can be true. Per-token prices fall while tokens per task rise, and nobody outside the providers sees the underlying cost.

Measurement is the second problem. Provider disclosures are self-reported and set their own boundaries, and Doctorow argued that hardware is depreciated over five years but replaced in two or three, without naming a company. Nvidia researchers argued in June 2025 that 7-billion-parameter models cost 10 to 30 times less to serve than models of 70 to 175 billion parameters, suggesting that much current spending buys capability many tasks do not need. Buyers also struggle to see their own costs. KPMG found only 35% of organisations had full visibility of AI operating costs, and 49% had scaled back or paused agent deployments because costs outweighed value.

Not the same as

Training cost is the periodic expense of teaching a model its parameters. It is large but fixed; inference cost recurs with every request.

API price is what a provider charges per token, which may sit above or below its actual serving cost.

Capital expenditure covers data centres and chips bought in advance. It funds the capacity inference consumes but is depreciated over years.

Energy footprint measures watt-hours, carbon and water per prompt. It is one input into inference cost, not the cost itself.

Recent developments

October 2026 brought price moves in both directions. Anthropic's Haiku 5.5 cut small-model rates by 90% for most prompts, while Google withheld its larger models from free Gemini users and opened Gemini 4 Argon to paying customers only. ChatGPT ads have reached 40 countries even as CPMs fell. The open question is whether ad revenue per answer can rise faster than the cost of producing answers as reasoning and agents push tokens per request higher.

Timeline

  • November 30, 2022 - OpenAI releases ChatGPT; days later Sam Altman calls its compute costs "eye-watering."
  • February 23, 2023 - Reuters reports John Hennessy's estimate that an LLM exchange costs about 10 times a keyword search; Morgan Stanley puts a 2022 Google search at about a fifth of a cent.
  • November 12, 2024 - Andreessen Horowitz's Guido Appenzeller estimates the price of equivalent LLM capability is falling about 10 times a year.
  • January 2025 - Altman says OpenAI is losing money on its $200-a-month Pro subscriptions.
  • February 9, 2025 - Altman's "Three Observations" post says the cost of a given level of AI falls about 10 times every 12 months.
  • March 12, 2025 - Epoch AI finds price declines of 9 to 900 times a year depending on the task.
  • June 2, 2025 - Nvidia Research publishes "Small Language Models are the Future of Agentic AI," claiming 10 to 30 times lower serving cost for 7-billion-parameter models.
  • August 21, 2025 - Google reports 0.24 watt-hours for the median Gemini text prompt and a 33-fold energy reduction over 12 months.
  • February 4, 2026 - Anthropic pledges to keep Claude ad-free.
  • February 9, 2026 - ChatGPT ads launch in the US at a $60 CPM.
  • April 2026 - Alphabet guides 2026 capital expenditure of $175 billion to $185 billion; ChatGPT CPMs reported as low as $25.
  • June 24, 2026 - KPMG reports 49% of organisations scaled back or paused AI agent deployments over costs.
  • August 31, 2026 - OpenAI says ChatGPT ads passed a $1 billion annualised run rate.
  • October 7, 2026 - Anthropic releases Haiku 5.5 at $0.10 input and $0.50 output per million tokens.
  • October 9, 2026 - Free Gemini accounts are limited to the Flash-Lite model.
  • October 13, 2026 - Google prices Gemini 4 Argon at $2 input and $10 output per million tokens.

Summary

Who: Model developers such as OpenAI, Google and Anthropic, and the cloud providers behind them, carry inference cost directly. Platforms building AI into products, agencies running agents and advertisers funding free chatbot tiers carry it indirectly through API fees, compute contracts and ad spend.

What: Inference cost is the recurring compute and money cost of running a trained AI model on each request, covering chips, energy, memory and idle capacity. It is usually sold per million input and output tokens and experienced as cost per query.

When: The term entered business use after ChatGPT's release on November 30, 2022. Per-token prices fell steeply from 2023 to 2026, while reasoning models and agents raised the number of tokens per task. In October 2026, providers cut small-model prices and restricted larger models to paying users.

Where: The cost is incurred in data centres running GPUs and custom accelerators, and surfaces in API price lists, consumer subscription tiers, agency token budgets and the ad products now appearing inside chatbots.

Why: Unlike training, inference cost grows with every user and every answer. It determines whether free AI products can be sustained, why chatbots are adding ads, how many ads each answer must carry and whether AI agents cost more to run than the media work they replace.