A verified bot is an automated client, such as a search crawler, a link-preview fetcher or an AI agent, whose claim to belong to a named operator has been checked and confirmed. The check matters because a bot announces itself through its user agent, a free-text header that any program can copy. Verification ties that claim to something harder to fake: the network addresses the operator controls, the domain names those addresses resolve to, or a private signing key. The concept exists so that site owners, security vendors and ad measurement firms can let legitimate automation through while challenging or blocking everything that merely says it is legitimate.

How verification works

Three mechanisms dominate, and they are often layered.

The oldest is forward-confirmed reverse DNS. A server takes the IP address of an incoming request and looks up the hostname registered to it. For Google's main crawler that hostname should end in googlebot.com, google.com or googleusercontent.com, according to Google's crawler documentation. The server then performs a forward lookup on that hostname and checks that it resolves back to the original address. Both steps are needed: anyone can set a misleading reverse record on an address they control, but they cannot make Google's domain point to it. Microsoft documented the same method for its crawler in August 2012, with hostnames ending in search.msn.com, as covered in the bingbot explainer.

The second is published IP ranges. Operators post machine-readable lists of the address blocks their bots use, and verifiers match incoming requests against them. Bing released a bingbot.json file on November 15, 2021, reversing its 2012 position that it did not publish crawl ranges. Google maintains separate JSON files for common crawlers, special-case crawlers such as AdsBot and user-triggered fetchers, and in March 2025 moved their refresh from weekly to daily. Anthropic, OpenAI, Perplexity and Amazon publish comparable lists.

The third is cryptographic signing. Under Web Bot Auth, a profile of HTTP Message Signatures (RFC 9421, published February 2024), the bot signs each request with a private key and attaches Signature, Signature-Input and Signature-Agent headers. The verifier fetches the operator's public keys from a well-known directory and checks the signature. Identity then travels with the request rather than with the network, which matters for agents running on shared cloud infrastructure where IP ranges prove little.

Verification by any of these methods produces the same practical output: a flag on the request. At Cloudflare that flag is exposed to customer firewall rules as the Boolean field cf.bot_management.verified_bot, which, according to Cloudflare's documentation, indicates "whether the request originated from a known good bot or crawler" and requires an Enterprise plan with Bot Management. A lower-tier equivalent, cf.client.bot, carries the same information. Site owners then write rules that skip challenges for flagged traffic, rate-limit it or block specific categories.

Programmes and directories

Identity alone does not make a bot verified in the commercial sense. Infrastructure providers run verification programmes that add behavioural conditions. Cloudflare's Verified Bots Policy, the most widely cited, requires more than 1,000 requests per day across multiple domains, a user agent of at least five characters that does not duplicate another verified service, publicly documented behaviour, and crawling only with the explicit or implicit consent of site owners. Search crawlers must read robots.txt, and the policy states that "the service should have a rate limit of 1 request per second." Scraping, scalping, credential stuffing and DDoS tooling are excluded outright.

Cloudflare's current documentation frames the bar as two tests: honest self-identification and non-abusive behaviour. Breach of either can remove a service from what Cloudflare calls the "global allowlist". Submissions now pass through BotBase, its bot directory, where an automated review fetches the submitted IP list, confirms reverse DNS or validates a Web Bot Auth signature before a bot is tracked. Annual submissions have grown roughly sevenfold since 2023, according to Cloudflare.

Each verified bot is also assigned categories. Since July 1, 2026, Cloudflare has classified bots by behaviour - Search, Agent, Training, Transact, SEO, Ads Verification and others - and by operator type, "Direct" for a single operator on its own infrastructure and "Intermediary" for agentic services that many end users can drive.

Origin and evolution

The problem verification solves is as old as commercial crawling. On September 21, 2006, Google's Matt Cutts published an official method for authenticating Googlebot after site owners at the SES San Jose conference asked for one. The reverse-then-forward DNS check described then remains Google's default recommendation two decades later.

Advertising measurement built its own parallel system. On February 7, 2006, the IAB and ABC Electronic announced a merged International Spiders and Bots List, a subscription list of known non-human user agents used to strip robotic activity from impression and traffic counts. The announcement estimated non-human activity at up to 40 percent of ad impression counts. The Media Rating Council's invalid traffic guidelines, finalised on October 27, 2015, made list-based filtering of declared bots part of general invalid traffic detection.

Cloudflare formalised the security-side version. Its March 16, 2022 announcement described manual verification using IP lists, reverse DNS, user agent and autonomous system validation, and machine learning, with approvals taking "a few weeks". It also began publishing the verified list publicly through Cloudflare Radar.

The AI crawler surge forced the next change. On May 15, 2025, Cloudflare proposed Web Bot Auth, arguing that IP addresses should stop serving as identity. Message signatures joined the Verified Bots Programme on July 1, 2025, and a separate signed agents category followed on August 28, 2025. That category has since been folded back in; Cloudflare's documentation now states that "Signed agents are now Verified." The IETF working group adopted the signing protocol as a working group draft on September 1, 2026, and in May 2026 Google documented Web Bot Auth as an experimental verification method for Google-Agent.

Why it matters for marketers

Verification decides which automation reaches content and ads. Bots made up 57.4% of HTML traffic across Cloudflare's network in early June 2026, according to Cloudflare telemetry cited when the company tied crawler access to four disclosure conditions. With traffic at that level, an allow rule for verified bots is often the only thing standing between a search crawler and a challenge page. Landing-page crawlers such as Google AdsBot need the same pass, because blocked ad-quality checks can disapprove ads.

Impersonation shows why the label carries weight. In August 2026 GreyNoise traced scanners forging the names of 13 AI crawlers from 824 addresses while hunting for credential files. None of the addresses matched any published crawler range, so range checks alone would have exposed them. IAB Australia's guidance, published July 28, 2026, treats unverified user agents as a category to challenge or block and recommends matching user agent, source address and behaviour together.

For media buyers, verified and declared bots should never receive paid impressions. A March 2025 Adalytics report found that ads were served to declared bots on known data-centre addresses despite verification vendors being in place, partly because some buying platforms did not pass the IP and device data needed to identify them. DoubleVerify called the report "misleading and manufactured".

Limitations and disputes

Verification proves who operates a bot, not what it does with the content. A crawler can be fully verified and still feed model training that a publisher objects to, which is the problem posed when, as with Googlebot, search and AI uses share one identity.

Verified status is also revocable, and revocation is a judgment by a private company. In August 2025 Cloudflare de-listed Perplexity as a verified bot after documenting 3 to 6 million daily requests from an undeclared crawler impersonating Chrome on macOS, alongside 20 to 25 million from the declared one. Perplexity disputed the characterisation. Critics in the IETF process argue that identity-based access control favours well-resourced operators and concentrates gatekeeping in firms that already sit in front of much of the web; Cloudflare alone fronts roughly a fifth of websites.

Each method has technical gaps. Reverse DNS fails for user-triggered fetchers on shared cloud hosts. IP lists go stale between refreshes. Signatures can be replayed until they expire, and Web Bot Auth has no revocation mechanism beyond removing a key and waiting for caches to refresh. Implementations also differ: Cloudflare and Google send incompatible forms of the Signature-Agent header.

Not the same as

Good bot is a looser, behavioural label for automation considered beneficial. Most verified bots are good bots, but many good bots, such as small monitoring tools, never apply for verification.

Known bot usually means a bot recognised by its user agent, for example on the IAB/ABC list, without proof that the request actually came from that operator.

Invalid traffic is the advertising measurement category for impressions and clicks that should not be billed. Under MRC rules, traffic from declared and verified crawlers is general invalid traffic to be filtered, so verification and invalidity are complementary, not opposites.

Verified purchase is an e-commerce review label, used notably by Amazon, confirming that a reviewer bought the product. It has no connection to bot identity.

Recent developments

The pace of change accelerated through 2026. Cloudflare's BotBase opened to operators on August 28, 2026, with submission history and automated checks. On September 15, 2026, Cloudflare began blocking Training and Agent crawlers by default on ad-carrying pages for newly onboarded domains, which makes accurate classification of verified bots directly consequential for publisher revenue. Google, meanwhile, signs only some Google-Agent requests and still recommends IP and DNS checks as the default, leaving verification in a hybrid state where network-based and cryptographic identity coexist.

Timeline

  • February 7, 2006: IAB and ABC Electronic announce the merged International Spiders and Bots List
  • September 21, 2006: Google publishes its reverse and forward DNS method for verifying Googlebot
  • August 2012: Microsoft documents reverse DNS verification for bingbot
  • October 27, 2015: MRC finalises its Invalid Traffic Detection and Filtration Guidelines
  • November 15, 2021: Bing publishes the bingbot.json IP range file
  • March 16, 2022: Cloudflare describes its verification methods and publishes verified bots on Radar
  • February 2024: RFC 9421 HTTP Message Signatures published
  • March 2025: Google moves crawler IP range files to daily refreshes
  • May 15, 2025: Cloudflare proposes Web Bot Auth
  • July 1, 2025: Message signatures accepted in Cloudflare's Verified Bots Programme
  • August 4, 2025: Cloudflare documents undeclared Perplexity crawling and de-lists it as verified
  • August 28, 2025: Cloudflare adds a signed agents category
  • May 4, 2026: Google documents Web Bot Auth as experimental verification for Google-Agent
  • July 1, 2026: Cloudflare introduces behaviour taxonomy and folds signed agents into Verified
  • July 28, 2026: IAB Australia publishes bots and crawler guidance
  • August 28, 2026: Cloudflare opens BotBase to bot operators
  • September 1, 2026: IETF working group adopts the Web Bot Auth protocol draft
  • September 15, 2026: Cloudflare default blocking of Training and Agent crawlers on ad-carrying pages begins

Summary

Who: Bot operators such as Google, Microsoft, OpenAI and Anthropic, which publish IP ranges, DNS records and signing keys; infrastructure providers such as Cloudflare, which run verification programmes; and measurement bodies such as the IAB and MRC, which require declared bots to be filtered from ad counts.

What: An automated client whose claimed identity has been confirmed through forward-confirmed reverse DNS, published IP ranges or HTTP message signatures, and which in commercial programmes must also meet behavioural rules on volume, documentation and robots.txt compliance.

When: Google published the first widely used method on September 21, 2006. Cloudflare formalised its programme publicly in March 2022, and cryptographic verification arrived from May 2025, with IETF adoption on September 1, 2026.

Where: At the network edge, in CDN and firewall rules, in server log analysis, and in ad verification and invalid traffic filtering systems.

Why: User agents can be copied by anyone, and automated traffic now exceeds human traffic on large networks. Verification lets sites admit search, ad-review and agent traffic while blocking impostors, though it confirms identity rather than intent, and the power to grant or revoke it rests with a small number of companies.