Reinforcement learning is a branch of machine learning in which software learns by acting, observing what follows, and adjusting. There are no labelled correct answers. An agent reads the state of an environment, chooses an action, receives a numerical reward, and revises the rule it uses to pick the next action so that reward accumulated over time rises. It exists because a large class of problems offers nothing to imitate: nobody can label the correct bid for one advertising impression, or the correct next sentence in a conversation, but both can be scored afterwards.

Three properties separate it from the rest of machine learning. Feedback is evaluative rather than instructive: it says how good an action was, not what the better action would have been. Rewards arrive late, so credit must be traced backwards across earlier decisions. And the agent's own behaviour determines the data it sees next, creating a permanent trade-off between exploiting what works and exploring what might work better.

How the loop works

The standard formalism is the Markov decision process, or MDP: a set of states, a set of actions, a transition rule describing how the environment moves between them, and a reward function. A policy maps states to actions. A value function estimates total future reward expected from a state, discounted so near-term reward counts for more than distant reward.

Methods divide along two axes. Model-free approaches learn a policy directly from experience; model-based approaches first learn how the environment behaves, then plan against that model. On-policy learning improves the policy currently running; off-policy learning improves a different one from data collected elsewhere, which is what permits training on historical logs.

Temporal-difference learning underpins most of it. Rather than waiting for an episode to end, the agent updates its estimate of one state's value using its own estimate of the next state's value, a bootstrapping step that lets learning run continuously. Q-learning, an off-policy variant introduced by Chris Watkins in 1989, learns the value of a particular action in a particular state. Policy-gradient methods instead nudge a policy network's parameters in the direction that raises expected reward. Proximal Policy Optimisation, published by OpenAI researchers in July 2017, limits how far a single update may move the policy, and became the default because it stays stable at scale.

Where it sits in a bidding system

Auto-bidding is the clearest advertising application. An advertiser sets a budget and a constraint, typically a cost-per-click ceiling or a return-on-ad-spend floor, and the platform decides how aggressively to bid across a period usually running 24 hours or longer.

Alibaba researchers set out the standard formulation at ACM SIGKDD in August 2024. The state at each time step holds five variables: remaining time, remaining budget, budget spend speed, real-time cost-efficiency and average cost-efficiency. The action adjusts the bidding parameters, the multipliers converting a predicted impression value into a submitted bid. The reward is the value contributed to the objective in that period.

The level at which the agent operates is easily misread. It is not pricing each impression, which a separate prediction model handles in the milliseconds before an auction closes. It steers the parameter scaling all of those prices, closer to a trader turning a dial than to one quoting each lot.

Training happens away from the live system. The same paper records that learning through direct interaction with the online environment "is unfeasible due to safety concerns", so policies are fitted to historical logs and only then deployed. That constraint shapes most of what goes wrong later.

Origin and evolution

The theoretical base is Richard Bellman's work on dynamic programming and Markov decision processes in the 1950s. The modern field dates to papers by Andrew Barto and Richard Sutton from the 1980s, which introduced the main ideas and produced the core algorithms. Their 1998 textbook has been cited more than 75,000 times, according to ACM.

Games supplied the proof. The decisive demonstration arrived on 26 February 2015, when a DeepMind team published a Nature paper describing a deep Q-network that learned 49 Atari 2600 games from raw pixels and the on-screen score, matching a professional human tester with one architecture and one set of hyperparameters. AlphaGo followed in 2016.

Advertising research moved quickly: real-time bidding by reinforcement learning at WSDM in 2017, budget-constrained bidding by model-free reinforcement learning at CIKM in 2018, and Alibaba's sustainable online reinforcement learning for auto-bidding at NeurIPS in 2022.

Language models then changed what the term means in public. Reinforcement learning from human feedback, or RLHF, fits a reward model to human preference comparisons, then optimises a language model against that proxy. DeepSeek released R1 on 22 January 2025, reporting reasoning behaviour that emerged from large-scale reinforcement learning against rule-based rewards. Six weeks later, ACM named Barto and Sutton recipients of the 2024 Turing Award.

Why it matters for marketers

Exploration has become an exposed control rather than a hidden one. Google's Smart Bidding Exploration lets an advertiser accept a lower effective return-on-ad-spend target so the system will bid on queries it would otherwise skip. The tolerance runs from 5% to 30%, and Google's documentation gives the arithmetic plainly: a 200% target with a 10% tolerance produces an effective average target of 180%. Google reported an 18% increase in unique search query categories producing conversions and a 19% increase in conversions overall, on internal data covering 11 March to 11 April 2025. The company expanded it to Performance Max and Shopping on 15 June 2026, having signalled it in Mayamong 42 Marketing Live launches. Product managers Kristina Park and Carlo Buchmann called it the biggest advancement in bidding in years; Google reinstated Target CPA and Target ROAS as standalone labels that same month.

Ranking systems use the same machinery. Meta finance chief Susan Li disclosed that user-understanding improvements combined with the GEM ranking system produced an 8.3% increase in advertisement clicks and a 15.7% uplift in conversions on Facebook in the second quarter of 2026, against a 3.5% click lift two quarters earlier. Measurement vendors use it too: Smartly signed a letter of intent on 16 March 2026 to buy INCRMNTAL, which applies causal inference and reinforcement learning to estimate incremental value without user-level data.

Limitations and disputes

Reward misspecification is the central failure mode. An agent optimises the reward it is given, not the outcome that reward was meant to represent. The clearest documented case sits in language models: an MIT paper submitted to arXiv on 22 February 2026 attributed chatbot sycophancy to RLHF, on the grounds that agreeable answers earn positive ratings, and found that half of simulated conversations with an always-sycophantic bot ended in 99% confidence in a false belief, an effect that restricting the model to true statements narrowed without closing. RLHF is also the layer safety filters depend on, and those filters failed on X in early 2026.

The offline-to-online gap is the bidding equivalent. Policies trained on logged data meet a live market whose competitors have changed behaviour since. Alibaba researchers named the problem inconsistency between online and offline in their 2022 NeurIPS paper and proposed training against the real system instead.

The formalism itself is contested. The same group argued in 2024 that the Markov assumption does not hold in auto-bidding, because the next bidding state correlates with a long history rather than the last state alone, and that Bellman-equation methods compound errors when returns are sparse. Their diffusion-based alternative reported a 2.81% gross merchandise volume gain over reinforcement learning baselines.

Evidence quality is the practitioner-facing dispute. The Trade Desk said an upgraded optimisation model cut cost per acquisition 32% across 62 campaigns in August 2026, comparing two model versions rather than testing against a control, and published no methodology document. Consultant Joey Bidner argued that Google's 17 August 2026 change to honour stated targets more strictly would strip the slack he relied on to reach new customers; Google denied the change was broader than described.

What it is not

Supervised learning trains on labelled examples and receives instructive feedback. Click-through-rate and conversion-rate prediction, the models that value an impression, are supervised. Reinforcement learning sits above them, deciding what to do with the output.

Generative AI describes what a model produces. Reinforcement learning is one method used to shape that production, applied after pre-training rather than instead of it, as covered in PPC Land's explainer on AI marketing.

A/B tests and incrementality experiments measure a difference between fixed conditions and stop. Exploration inside a reinforcement learning system is continuous and feeds straight back into the policy, a distinction drawn out in PPC Land's incrementality explainer.

Agentic AI delegates multi-step workflows to tool-calling language-model agents, a different architecture described in PPC Land's agentic AI explainer, though reinforcement learning often trains the underlying model.

Recent developments

Research is moving from per-scenario agents to general ones. Alibaba's Bid2X, posted to arXiv on 27 October 2025, replaces scenario-specific bidding models with one foundation model; two months of A/B testing on Taobao raised gross merchandise volume 4.65% and return on investment 2.44% against the platform's existing model-based reinforcement learning system.

Platform controls are being reshaped rather than removed. The Trade Desk replaced Koa's on-off switch with three optimisation modes in closed beta on 30 April 2026, leaving the system on by default, after forced Kokai adoption drew resistance from agency trading desks. Meta described a further shift in the same quarter, the Meta Generative Recommender, which reasons jointly about advertisement content and user preference rather than scoring each candidate separately, and reported Advantage+ passing a $75 billion annual run rate. PPC Land's Advantage+ explainer traces how that layer determines delivery.

Timeline

  • 1957: Richard Bellman formalises dynamic programming and Markov decision processes
  • 1980s: Andrew Barto and Richard Sutton publish the papers establishing modern reinforcement learning
  • 1989: Chris Watkins introduces Q-learning
  • Early 1990s: Gerald Tesauro's TD-Gammon reaches expert backgammon play
  • 1998: Sutton and Barto publish Reinforcement Learning: An Introduction
  • 26 February 2015: DeepMind publishes the deep Q-network Nature paper covering 49 Atari games
  • March 2016: AlphaGo defeats Lee Sedol
  • July 2017: OpenAI publishes Proximal Policy Optimisation
  • 2017: Real-time bidding by reinforcement learning presented at WSDM
  • 2018: Budget-constrained bidding by model-free reinforcement learning presented at CIKM
  • 2022: Sustainable online reinforcement learning for auto-bidding presented at NeurIPS
  • August 2024: Alibaba presents diffusion-based generative auto-bidding at ACM SIGKDD
  • 22 January 2025: DeepSeek releases R1, trained with Group Relative Policy Optimisation
  • 5 March 2025: ACM names Barto and Sutton recipients of the 2024 Turing Award
  • 27 October 2025: Bid2X posted to arXiv
  • 6 November 2025: Smart Bidding Exploration appears in Google Ads Editor 2.11
  • 22 February 2026: MIT sycophancy paper submitted to arXiv
  • 30 April 2026: The Trade Desk opens closed beta of three Koa optimisation modes
  • 15 June 2026: Google expands Smart Bidding Exploration to Performance Max and Shopping
  • 17 August 2026: Google's bidding target optimisation change takes effect

Summary

Who: Researchers at DeepMind, OpenAI, Alibaba, DeepSeek and university groups develop the methods. Advertising platforms including Google, Meta, The Trade Desk and Amazon deploy them inside bidding and ranking systems. Advertisers and agency traders set the objectives and constraints those systems optimise against. Andrew Barto and Richard Sutton established the foundations and received the 2024 ACM Turing Award for it.

What: A branch of machine learning in which an agent learns a policy by acting in an environment and receiving numerical rewards, rather than by imitating labelled examples. In advertising it steers bidding parameters across a campaign period; in language models it aligns outputs to a reward model fitted on human preferences.

When: Theoretical foundations date to the 1950s, the modern field to Barto and Sutton's papers from the 1980s. Deep reinforcement learning arrived with the February 2015 Nature paper on Atari. Advertising applications entered the literature from 2017. RLHF became the standard alignment method from 2022, and reasoning models trained largely by reinforcement learning from January 2025.

Where: Inside auto-bidding engines at Google, Meta, Alibaba and demand-side platforms; inside recommendation and ranking systems; inside the post-training stage of every major commercial language model; and, increasingly, inside measurement products estimating incremental value.

Why: Impression-level pricing and conversational response both lack labelled correct answers but can be scored after the fact, which is exactly the shape reinforcement learning addresses. The costs are documented alongside the gains: agents optimise the reward supplied rather than the outcome intended, offline training diverges from live markets, and platform-reported improvements are frequently model-versus-model comparisons rather than controlled tests.