Cloudflare on October 16 published the outline of a system that watches how its Verified bots behave after they have been approved, measuring whether each one obeys robots.txt rules and slows down when a website returns an HTTP 429 "Too Many Requests" response. The observations will be turned into what the company calls a trust band, which it plans to publish on Radar and, later, to let customers use as a threshold for deciding how Verified bot traffic is handled.

In Short

Cloudflare keeps a list of approved bots, and it will now keep checking whether those bots follow the rules websites set, instead of approving them once and moving on. That matters because publishers and advertisers rely on "verified" status to decide which crawlers get in, and a bot that ignores "do not crawl" or "slow down" signals can cost them server capacity and content. Bots that keep ignoring those signals will slip down a trust ranking, and Cloudflare customers will eventually be able to choose how low a bot can fall before its traffic is treated differently.

What Cloudflare published

The post, titled "Continuous accountability for Verified bots" and written by Chris Tsoukaladelis, appeared on the Cloudflare Blog on October 16, 2026. It is short - listed as a six-minute read - and it describes a system that is already running inside the company rather than a product customers can switch on. The post carries the tags AI, Bots, Internship Experience, Radar and Security.

Its framing starts with an analogy. "A driver's license tells you who someone is and that they passed a driving test, not whether they'll drive responsibly in the future," according to Cloudflare. "The same is true for bots: verifying who operates it creates accountability, but trust depends on what it does next."

The verified bot program, as Cloudflare describes it, has always asked two questions: "Are you who you say you are?" and "Can we trust you?" Bot operators broadly have to satisfy three criteria to keep the label. They must identify themselves accurately and securely, do only what they say they will do, and not engage in abusive behavior. Identity, the first criterion, is handled by Web Bot Auth, the cryptographic signing scheme bots use to establish who is sending a request, which Cloudflare's BotBase then catalogues in an open format. The new system targets the third criterion. In the company's words, the focus "shifts to the second cardinal rule of Verified bots: thou shalt do no harm."

The new system is designed "to evaluate and report when Verified bot behavior starts to deviate from this rule, ensuring that the answer to 'Can we trust you?' remains 'Yes,'" according to the post.

Risk versus trust

Cloudflare draws a line between two concepts that its bot products have often blended. Risk, according to the company, "describes the likelihood that a particular request or point-in-time action may be harmful, and it can change from one moment to the next." Trust, by contrast, "is built over time through repeated and consistent behavior."

The distinction explains why the new system does not act on single events. A bot score attached to one request measures risk. A trust band attached to an operator over weeks or months measures something else - whether a bot that has already passed identity checks continues to behave the way it promised.

Cloudflare points to one precedent. "We de-listed Perplexity's bot from our Verified bot list in 2025 when it sought to access resources on websites that opted out of scraping and subsequently attempted to circumvent controls when those requests were blocked," according to the post. PPC Land covered the episode when Cloudflare documented stealth crawling by Perplexity, including the use of a generic Chrome-on-macOS user agent after the declared crawler was blocked. That removal was a one-off decision. What the October 16 post describes is a standing mechanism that could, in principle, catch the same behavior earlier and on a graded scale rather than as a binary delisting.

How the trust band works

The output of the system is a trust band - described as "a range showing how consistently a bot respects site preferences and which observed behaviors informed that assessment, rather than a permanent judgment about a bot or its operator." Two design choices stand out in that sentence. The band is a range, not a single number, and it is meant to be explainable, with the behaviors behind it visible.

Two initial signals feed it:

  • Does the bot respect instructions?
  • Did it respond to direct feedback?

Cloudflare calls these "common sense" rules and "a courtesy to the operators who provide the information bots seek." The company says it will expand "the ways we measure" in the future, but the post does not name any additional signals. Nor does it say how many bands exist, what separates one band from the next, how long an observation window is, or how many violations move a bot down. Those omissions matter, because the eventual customer feature depends on a threshold that has not been described.

Signal one: robots.txt

The first signal concerns robots.txt, which Cloudflare calls "the primary way to communicate site owner preferences." The system "considers whether a bot's observed traffic is consistent with those published preferences over time."

Cloudflare is explicit that it is not hunting for isolated mistakes. "Isolated violations and other types of one-offs may not be significant," according to the post. "What we look for instead are repeated patterns of behavior that disregards site preferences." The expected standard is RFC 9309, the IETF document that formalized the Robots Exclusion Protocol in 2022: responsible bots are expected to "consult and honor the preferences that websites publish."

The post also makes an argument about what robots.txt does not do. "A robots.txt sets the minimum rules for what bots and agents should not do. But there is no implication that anything not stated is allowed," according to Cloudflare. A site does not need to write a rule telling a bot to stop if it can signal through other channels that continued access is unwelcome. That position closes a gap that some crawler operators have relied on - the reading that any path not disallowed is fair game.

Signal two: HTTP 429

The second signal addresses load rather than access. "Bots can remain entirely within allowed paths while still placing substantial load on the websites they visit," the post states. One of the clearest ways a site can push back is an HTTP 429 response, the status code for "Too Many Requests," defined in RFC 6585. A 429 is, according to Cloudflare, "a directive for the visitor to reduce its request rate, sometimes with guidance about how long to wait before trying again."

The system watches what happens next. Does the bot slow down, pause, or keep sending requests at the same pace? "Consistently slowing down or pausing increases trust, while repeatedly ignoring the signal lowers it," according to the post. This is the only place in the document where the direction of movement within a band is spelled out.

Load is not an abstract concern. The Wikimedia Foundation has said OpenAI agents may have contributed to a partial outage of the Wikidata Query Service, with bots accounting for 65% of its most resource-consuming traffic. A crawler that obeys robots.txt to the letter can still cause that kind of strain, which is why Cloudflare treats the two signals separately.

The accountability loop

Cloudflare positions the new system as the middle piece of three. Bot Preference Sync, which generates or updates a site's robots.txt from the AI crawler policy set in the dashboard, handles the publishing side. Bot Management handles enforcement. The trust system observes whether Verified bots honor what was published. "Together, they help ensure that trust is earned through transparent, accountable behavior, not assumed from identity alone," according to the post.

That structure links the October 16 announcement to a run of changes Cloudflare made over the summer. In August, the company said it would block opaque AI crawlers from sites that disallow training and detailed Bot Preference Sync, which prepends a generated section to an existing robots.txt between comment markers and is available on all plans. Days later, Cloudflare said annual bot submissions had grown roughly seven times since 2023 and that review of new Verified bot applications had become automatic, with checks for duplicate entries, user-agent specificity and the validity of IP lists, reverse DNS or Web Bot Auth signatures. That August update said Cloudflare increasingly determines whether sites can allow a crawler based on its behavior, but it described no mechanism for monitoring behavior after admission. The October 16 post fills that gap, at least in outline.

Automated admission and continuous re-checking fit together. If review of new bots is now largely machine-driven and volume keeps rising, a one-time approval becomes weaker evidence over time. "Verify, but continuously re-verify," the post says.

What is live and what is not

The system is internal for now. "We are already using this system internally to evaluate how consistently Verified bots honor website preferences," according to Cloudflare. The company describes the current period as an "initial phase" used to build behavioral history and "refine how those observations become useful guidance."

Two future steps are named, neither with a date:

  • Public scores on Radar. "In the future, we will be sharing these scores publicly on Radar, as well as expanding the ways we measure," according to the post. Cloudflare Radar is the company's public data site, which already publishes bot shares and AI crawler activity drawn from its network.
  • Customer thresholds. Cloudflare plans "to make the system available to all Cloudflare customers, who will be able to choose the trust threshold used to decide how Verified bot traffic is handled."

The second step is the one with operational consequences. At present, a site using Cloudflare can generally allow Verified bots as a category. A threshold would let a publisher admit only Verified bots above a certain band - so the label alone would no longer guarantee access. The post does not say what "handled" will mean in practice: blocking, challenging, rate limiting or logging are all possible, and none is specified.

Why this matters for publishers and marketers

For publishers, the question of which bots honor robots.txt has moved from theory to measurement over the past year. Microsoft Clarity in June began flagging robots.txt violations inside Bot Analytics, with an illustrative violation rate of 4.56% across 246,035 AI crawler requests. TollBit reported that 15% of AI page fetchers in Europe reached disallowed URLs in the first half of 2026. HasData found that 234 of 592 sites (39.5%) disallowing GPTBot still served it a live page, and that by September 16 none of 118 sites carried Cloudflare's managed AI block in robots.txt after the company's September 15 change. Those tools measure compliance site by site. Cloudflare's version would aggregate it across a network that, by the company's own estimate, sits in front of more than 20% of the web.

There is a commercial dimension. Cloudflare sells the bot management products that enforce these preferences, and it is the party deciding who counts as verified and, now, how trustworthy each verified bot is. The September changes that stopped Cloudflare's planned Googlebot block for sites refusing AI training already relied on a Cloudflare-defined "Accountable" standard to decide which search crawlers stayed allowed. A trust band adds another layer of Cloudflare-defined judgment. The post presents it as explainable and not permanent, but until the criteria and band boundaries are published, operators and site owners have no way to check how a given rating was reached.

For advertisers, the relevance is less direct but real. Agencies and ad tech vendors run their own crawlers and agents - for brand safety scans, ad verification, price monitoring and SEO auditing - and many rely on verified status to reach Cloudflare-protected pages. A bot that is technically verified but poorly behaved could find that status worth less once customers can set thresholds. IAB Tech Lab's bot management guidance for publishers, opened for comment in May, framed access control as a spectrum from open access to graduated controls. A trust threshold is one form of graduated control.

The scale argument behind all of this keeps growing. Cloudflare chief executive Matthew Prince said in September that he expects bot traffic to reach about 1,000 times human volume within five years, after automated requests passed human requests on Cloudflare's network in May 2026. If that forecast is close, the number of bots asking for verified status will keep rising, and the question in Cloudflare's post - can a verified bot still be trusted next month? - will be asked of far more operators.

Open questions

The post leaves several points unresolved. It does not say which Verified bots are currently being evaluated, whether any have already moved down a band, or whether operators will be notified before a score appears on Radar. It does not explain how a bot can recover after a period of poor behavior, though the description of the band as "not a permanent judgment" implies recovery is possible. And it does not describe how the system will handle agents acting on a user's behalf, a category that some operators argue sits outside robots.txt altogether.

What the document does establish is a direction. Identity was the first hurdle Cloudflare set for automated traffic, and Web Bot Auth was its answer. Behavior over time is the second. The stated goal, according to Cloudflare, is "to keep website preferences meaningful, and ensure trust is earned through sustained behavior."

Timeline

Summary

Who: Cloudflare, in a blog post by Chris Tsoukaladelis, describing a system that evaluates bots on its Verified bot list. The changes affect bot and AI crawler operators, and the website owners, publishers and advertisers who use Cloudflare to decide which automated traffic to admit.

What: A system that observes whether Verified bots follow robots.txt rules (per RFC 9309) and slow down or pause after receiving HTTP 429 "Too Many Requests" responses, turning repeated patterns into an explainable trust band. Isolated violations are not the focus; sustained behavior is.

When: The post was published on October 16, 2026. The system is already running internally. Public scores on Radar and customer-selectable trust thresholds are planned, with no dates given.

Where: Across Cloudflare's network, which the company says sits in front of more than 20% of the web, with results eventually published on Cloudflare Radar.

Why: Cloudflare argues that verifying a bot's identity says nothing about how it will behave later, and that trust has to be re-earned through consistent conduct. The system completes what the company calls an accountability loop with Bot Preference Sync, which publishes site preferences, and Bot Management, which enforces them.