Bot management is the set of tools and policies a website or app uses to identify automated traffic and decide, request by request, what that traffic is allowed to do. It exists because a growing share of web requests no longer come from people. Some bots are wanted, such as search crawlers and the ad crawlers that check landing pages. Others scrape prices, test stolen passwords, hoard inventory or click on ads. Bot management tries to treat each by identity and intent rather than blocking everything that is not human.

Imperva's 2026 Bad Bot Report, published on April 29, 2026, put automated traffic at more than 53% of all web traffic in 2025, up from 51% a year earlier. Cloudflare's measurement is higher: its Radar data for the seven days ending June 5, 2026 showed bots at 57.4% of HTML requests, a figure covered in PPC Land's report that bots now outnumber humans on the web. The two figures use different networks and definitions.

How a bot management system works

Most commercial systems run at the network edge, inside a content delivery network (CDN), a reverse proxy or a web application firewall (WAF). Each request passes through signal collection, classification and policy.

Signal collection starts with what the request declares: the User-Agent header, and a source IP address whose network reveals whether traffic comes from a home, a mobile carrier or a data centre. Deeper signals include the TLS handshake fingerprint (the JA3 and JA4 methods hash how a client negotiates encryption), HTTP/2 settings, header order, cookie behaviour and request timing. Injected JavaScript adds browser evidence, such as headless browser traces or mouse movement.

Classification turns those signals into a verdict. Cloudflare's documentation, last updated in August 2026, describes a bot score from 1 to 99, where 1 means the request is almost certainly automated and 99 means it almost certainly came from a human. Scores from 2 to 29 are grouped as "likely automated" and 30 to 99 as "likely human", with a separate verified bot category for declared crawlers whose identity has been confirmed. Heuristics assign a score of 1 to deterministic hits, such as a request with no User-Agent at all, while a supervised machine learning model produces most scores between 2 and 99. Granular scores are reserved for Enterprise customers.

Policy is where the site owner decides what to do with each class. The usual actions are allow, log, rate-limit, challenge (an invisible JavaScript test or a visible CAPTCHA) or block, and rules can be scoped to paths such as login or checkout. Hosting firms now package these as presets, as when Kinsta added free bot protection to all WordPress plans in June 2026.

Identity verification handles good bots. Googlebot and bingbot are confirmed through DNS lookups or published IP ranges; since 2025, Web Bot Auth lets operators sign requests with HTTP Message Signatures (RFC 9421).

Publishers and retailers operate these systems, usually through vendors such as Cloudflare, Akamai, Imperva (owned by Thales), HUMAN, DataDome, Arkose Labs, F5 and Fastly. Advertisers rarely configure them but are subject to them: Google's AdsBot must reach a landing page for an ad to pass review.

Origin and evolution

Bot management grew out of older tools. The robots.txt convention, proposed in 1994, asked crawlers to respect a site's wishes but enforced nothing. The CAPTCHA (Completely Automated Public Turing test to tell Computers and Humans Apart) was formalised at Carnegie Mellon University in 2003, and Google bought reCAPTCHA in 2009. WAFs filtered malicious payloads but missed bots sending valid requests.

Dedicated vendors appeared in the early 2010s. Distil Networks, which its chief executive Tiffany Olson Kleeman said had been "leading the charge" since 2011, focused on scraping and account abuse. The Open Web Application Security Project (OWASP) built a shared vocabulary for the problem in its Automated Threats to Web Applications project, which lists 21 threat types, from carding (OAT-001) and scalping (OAT-005) to credential stuffing (OAT-008) and scraping (OAT-011). The US Better Online Ticket Sales (BOTS) Act of December 2016 outlawed software that circumvents ticket purchase limits.

Consolidation marked the move from point product to platform feature. Imperva announced on June 4, 2019 that it would acquire Distil, and Forrester analyst Amy DeMartine was quoted in the announcement predicting that "bot management will become the predominant application defense." Cloudflare described the machine learning behind its Bot Management product on May 6, 2020. That version exposed a 0 to 100 score to firewall rules as cf.bot_management.score and ran CatBoost gradient boosting models; today's 1 to 99 scale is a later revision. Turnstile, Cloudflare's puzzle-free CAPTCHA replacement, followed in September 2022.

Generative AI changed the question from "human or bot?" to "which bot, doing what, on whose terms?" Cloudflare began blocking AI crawlers by default for new customers on July 1, 2025, and PPC Land covered the same day's launch when Cloudflare introduced Pay Per Crawl. Web Bot Auth followed, and in October 2025 Visa and Mastercard adopted it for agent authentication, a step reported when Cloudflare partnered with both card networks to secure AI agent shopping.

Industry bodies arrived last. IAB Tech Lab's Content Monetization Protocols working group released draft bot management guidance on May 27, 2026, and PPC Land reported that IAB Tech Lab opened the guidance for public comment. PPC Land gives the comment deadline as June 25, 2026, while AdExchanger reported June 26. No final version had been located as of November 2026. In Australia, the local IAB went further: its July 2026 decision matrix sorts crawlers into five categories and four verdicts, as described in PPC Land's report that IAB Australia forces every crawler into one of four verdicts.

Why it matters for marketers

The connection to advertising runs in three directions. The first is fraud. Bots that load pages and click ads generate invalid traffic. Publisher-side bot management reduces that junk, but does not replace the filtration the Media Rating Council (MRC) requires of verification vendors.

The second is visibility. An aggressive bot policy can lock out the crawlers marketing depends on. Google warned on November 11, 2026 that blanket protection during a CDN migration can block Googlebot, and PPC Land reported that CDN bot blocks can drop products from Shopping and ads. In Google's video, Gary Illyes and Cherry Prommawin said "products disappear from Google Shopping and ads products stop working."

The third is commerce. As AI shopping agents begin to browse and buy, a bot rule written to stop scalpers can also stop a paying customer's assistant. Blend Commerce tested an AI shopping agent against 200 UK and US retailers in September 2026 and found that bot protection blocked it at 29 of them, including Home Depot, Best Buy and Tesco. The agent was an unrecognised automated browser, not a named AI company's agent.

Limitations and disputes

Detection is uneven. DataDome tested 21,491 high-traffic domains in June 2026 with 10 test bots and found that 65.3% stopped none, while only 2.4% stopped all ten, down from 8.4% in 2024. PPC Land covered the study when full bot protection dropped to 2.4% of popular websites. Resemble AI claimed in October 2026 that established tools caught between 3.1% and 14.8% of AI agents that hid their automation - PPC Land noted the missing sample size in Bot tools caught 3.1% to 14.8% of disguised AI agents. Both come from vendors selling competing products.

False positives add friction for humans, and visual CAPTCHAs have long drawn accessibility complaints. Fingerprinting and behavioural telemetry also raise privacy questions under EU consent rules.

Concentration is a third concern. When a handful of edge providers decide which bots may reach millions of sites, a single fault spreads widely. On November 18, 2025, a database permissions change caused the file feeding Cloudflare's Bot Management module to roughly double in size, according to Cloudflare's post-mortem. The file crashed the core proxy and broke a large part of the web for hours.

Finally, identity remains thin. Arkose Labs reported that only 2.3% of legitimate AI agent sessions in the third quarter of 2026 carried a verified Web Bot Auth signature, as covered in PPC Land's report that 97.7% of legitimate AI agent sessions carry no verified ID.

Bot management versus adjacent terms

  • Web application firewall: a WAF inspects request content for attacks such as SQL injection; bot management inspects who is sending it.
  • Invalid traffic (IVT) detection: removes non-human impressions and clicks from ad measurement, while bot management acts before content is served.
  • Verified bot: a label for declared bots whose identity is confirmed, an input to policy rather than a policy.
  • DDoS mitigation: absorbs distributed denial-of-service floods; bot management targets lower-volume automation that looks legitimate.

Recent developments

Cloudflare said on October 16 that verified bots would receive trust bands based on whether they honour robots.txt and slow down after HTTP 429 responses, a change PPC Land reported as Verified bots face trust scores for ignoring site rules. On October 1, DataDome joined Experian's Agent Trust ecosystem, adding behaviour monitoring to Experian's checks on AI shopping agents.

Cloudflare's November 5 paper "Agents aren't humans (but they kinda are)", by Bryan Becker, argued that sites bear most of the cost of agent traffic, a point PPC Land covered in its report that websites pay for AI agents. United Airlines revised its Contract of Carriage on November 11 so that it bans AI agents from buying its tickets, seats and upgrades. Cloudflare chief executive Matthew Prince said in September that automated traffic passed human traffic on Cloudflare's network in May 2026, and forecast bot traffic at 1,000 times human levels within five years, adding the caveat that "I've gone wrong so far in every prediction that I've made on this."

Timeline

  • 1994: robots.txt convention proposed for crawler etiquette
  • 2003: Carnegie Mellon researchers formalise the CAPTCHA
  • 2009: Google acquires reCAPTCHA
  • 2011: Distil Networks begins bot mitigation work
  • December 2016: US BOTS Act signed into law
  • June 4, 2019: Imperva announces its acquisition of Distil Networks
  • May 6, 2020: Cloudflare details the machine learning behind Bot Management
  • September 2022: Cloudflare launches Turnstile
  • May 2025: Cloudflare proposes Web Bot Auth
  • July 1, 2025: Cloudflare blocks AI crawlers by default for new customers and launches Pay Per Crawl
  • October 2025: Visa and Mastercard adopt Web Bot Auth for agent payments
  • November 18, 2025: Bot Management feature file triggers a global Cloudflare outage
  • April 29, 2026: Imperva puts automated traffic above 53% of web traffic in 2025
  • May 2026: automated traffic passes human traffic on Cloudflare's network, per Matthew Prince
  • May 27, 2026: IAB Tech Lab releases draft bot management guidance for comment
  • July 22, 2026: IAB Australia dates version 1.0 of its bots and crawler decision matrix
  • September 15, 2026: Cloudflare default-blocks Training and Agent crawlers on ad-bearing pages for new domains
  • September 22, 2026: DataDome finds 2.4% of popular sites stop all test bots
  • October 16, 2026: Cloudflare announces trust bands for Verified bots
  • November 11, 2026: Google warns CDN bot blocks can drop Shopping products; United bans AI agent bookings

Summary

Who: Website owners, publishers and retailers operate bot management, mostly through vendors such as Cloudflare, Akamai, Imperva, HUMAN, DataDome and Arkose Labs. Bot operators, from Google to AI companies, are subject to it, and IAB Tech Lab and IAB Australia have issued guidance.

What: A system that collects request signals, scores how likely each request is automated, confirms the identity of declared bots, and applies allow, challenge, rate-limit or block policies.

When: Dedicated products emerged in the early 2010s, consolidated from 2019, and were reshaped from 2023 by AI crawlers and, during 2025 and 2026, by AI shopping agents.

Where: At the network edge, inside CDNs, reverse proxies and WAFs, in front of websites, APIs and apps.

Why: Automated requests now exceed human ones by most measures, and they range from crawlers marketing depends on to fraud that drains ad budgets, so sites need to separate wanted automation from harmful automation without locking out customers or their agents.