Cloudflare on September 15, 2026 added a Disallow AI Training option to its bot controls, letting a site publish a no-training preference in robots.txt while the search crawlers run by Google, Apple and Microsoft keep indexing it, and in the same change turned its Block option into a switch that shuts those three crawlers out completely.

In Short

Cloudflare added a setting that tells the search crawlers from Google, Apple and Microsoft they can keep listing a site in search results but may not use its pages to train AI models. This matters because those three companies use one crawler for both jobs, so until now saying no to training risked saying no to search traffic as well. If you pick Cloudflare's stricter Block option now, those crawlers are cut off entirely, search included, and Microsoft will not read the new robots.txt rule until its support arrives, which is targeted for early 2027.

A fourth option between Allow and Block

The change was set out in a blog post titled "Have it both ways: stay discoverable in search while disallowing AI training," written by Bryan Becker, Director, Product Security at Cloudflare. Some of the largest companies on the internet operate what Cloudflare calls mixed-use crawlers: a single bot that gathers pages both for a search index and for model training. Googlebot, Applebot and Bingbot are the three that matter most.

Cloudflare classifies bots by behaviour, and one bot can show several. Three behaviours are exposed as domain-level controls: Search, meaning crawling to build a search index; Training, meaning crawling to train or fine-tune a model; and Agent, meaning user-directed software fetching a page on a person's behalf, such as chat fetch bots and browser-use agents.

With the September 15 update, the Training control carries four settings:

  • Allow lets every crawler through unless another setting or a firewall rule stops it.
  • Disallow AI Training has Bot Preference Sync write the applicable no-training preference into robots.txt. Mixed-use crawlers that Cloudflare labels Accountable stay allowed for search, and every other training crawler is blocked, including the training-only crawlers run by Amazon, Anthropic, Meta and OpenAI.
  • Block on pages with ads blocks crawlers, now including mixed-use ones, only on pages Cloudflare detects serving an ad.
  • Block stops all such crawlers, mixed-use ones included, across the domain.

Named for the Disallow directive it publishes, the new setting exists only for Training. The controls are available on every plan through a zone's Security Settings.

Cloudflare is candid about the limits of the file it is writing. "A robots.txt directive alone cannot solve this problem. Anyone can publish one, but it cannot identify who is crawling, determine why they are crawling, or stop a crawler that ignores it," the post states. Its answer is that the network publishes the preference, identifies and classifies the crawler, blocks those that ignore it, and reports what each operator does on Cloudflare Radar.

The post does not dwell on one consequence. Training-only crawlers are stopped at the edge, but for the three Accountable mixed-use crawlers the fetch still goes through, because search depends on it, and what happens to the page afterwards rests on the operator honouring the preference.

How the September 15 rule changed after July

The date was not new. On July 1, 2026, in a post titled "Your site, your rules," Cloudflare set September 15 as the start of new defaults under which Training and Agent crawlers would be blocked on ad-carrying pages for newly onboarded domains. The same post said multi-purpose crawlers would be judged by the most restrictive applicable rule, and named Googlebot, Applebot and BingBot as crawlers that would be blocked for customers who had chosen to block Training, whether through the new controls or the older Block AI bots toggle.

The documentation page supplied alongside the post still carries that earlier wording. Marked as last updated on July 1, 2026, it states: "Mixed-purpose crawlers that combine Search and Training will also be blocked by all configurations to block AI training, including the legacy 'Block AI bots' option." It lists three mitigation options, not four, and makes no mention of Disallow AI Training. The two documents describe different outcomes for the same sites; the later post reflects current behaviour.

A middle path first appeared on August 21, when Cloudflare published Bot Preference Sync, which generates robots.txt from dashboard settings, alongside a Disallow option for Training. The September post completes that turn. Block and Block on pages with ads, according to Cloudflare, "previously did not apply to mixed-use crawlers because blocking them could also affect search discoverability." Now they do, and existing Training blocks are moved to Disallow AI Training rather than being left to cut Google, Apple and Microsoft off.

The practical upshot is that removing those crawlers now takes a deliberate choice. "If you want mixed-use crawlers gone entirely, you now have to say so," the post states. For Bing, that choice carries weight beyond one search engine: blocking Bingbot takes a site out of Bing, Yahoo and Copilot at once.

Accountable, on the strength of commitments

The label that decides who keeps search access is Accountable. To earn it, an operator must meet, or commit to meeting, four requirements:

  1. A mechanism for site owners to opt out of AI training, through robots.txt or a similar standard.
  2. A mechanism for site owners to opt out of AI summaries, set with the operator directly and, from next year, through Cloudflare.
  3. URL-level visibility into which pages were made available for training, along with metrics showing how content appeared in search.
  4. Assurance that opting out of AI training will not affect traditional search results.

The list tracks the four conditions Cloudflare set out in August, but the wording has shifted. In August, operators had to supply additional information to avoid being blocked, and the fourth condition asked them to demonstrate publicly that disallowing training does not damage search. The September version accepts commitments alongside current capabilities, and asks for assurance rather than public demonstration.

That matters, because on Cloudflare's own account none of the three mixed-use operators meets every requirement in full yet. "Apple, Google, and Microsoft all demonstrate that they meet the qualifications to be Accountable," the post states, adding that each pairs capabilities already available with time-bound commitments for those still in development.

Cloudflare also classes the relevant crawlers from Amazon, Anthropic, Meta and OpenAI as Accountable, because those companies run separate search and training crawlers. Meta's arrangement is less tidy than that suggests. Meta-ExternalAgent carries a declared purpose covering both model training and indexing for Meta products, while a separate crawler, Meta-WebIndexer, builds the search index behind Meta AI. The post does not name which Meta crawler it treats as the training bot.

Google

Googlebot's training opt-out is a Disallow rule for Google-Extended in robots.txt, and Google offers a toggle in Search Console to exclude a site from generative search results, along with reporting on search and AI summary results. According to Cloudflare, Google shared details of further URL-level transparency tools tied to Google-Extended, which it expects in "the weeks to come." Google has stated that disallowing Google-Extended does not affect search ranking.

The scope of that token is narrower than the setting's name might suggest. Google added Google-Extended on September 28, 2023 as a control token with no user agent of its own. It governs training of Gemini models and grounding in Gemini Apps and Vertex AI, but not AI Overviews or AI Mode. Those are handled by the separate Search Console setting, which works at domain level, does not stop training and has no page-level option until March 3, 2027. A site on Disallow AI Training can therefore keep Gemini training off its pages while those same pages still feed Google's AI answers.

Apple

Applebot's opt-out is a Disallow rule for Applebot-Extended. Site owners can signal AI summary preferences through the nosnippet directive and label paywalled content to keep it out of generative output. Apple has no URL-level inspection tool yet; Cloudflare says Apple shared details of an in-progress solution for next year. Apple has stated that disallowing training does not affect search ranking. Applying nosnippet to AI output dates from Apple's June 8, 2026 rewrite of its Applebot documentation, and it works only at page level.

Microsoft

Bing is the gap. Microsoft currently takes AI training preferences through its NOARCHIVE meta tag and is building support for a domain-level robots.txt "no training" preference, targeted for early 2027. Until then, sites can combine NOARCHIVE with Bing's Block URLs or Content Removal tools; Microsoft has stated NOARCHIVE will not affect ranking. The post is explicit about the interim: "selecting Disallow AI Training will not automatically convey a no-training preference to Bing through robots.txt." Cloudflare describes this as the same practical behaviour as the previous Training Block, which never applied to Bingbot.

What happens to existing settings

"Nothing, in almost every case," is the post's answer to what customers need to do. A Cloudflare email to customers dated September 16, 2026 said settings would migrate automatically over the following week, that manual changes made in the meantime would be preserved, and that the migration is complete once the Block AI Bots switch disappears from the dashboard. The email describes the changes as taking effect on the day it was sent while also dating the shift to September 15, a one-day gap in Cloudflare's own framing.

For domains that only ever used the legacy toggle, the mapping is fixed. A disabled toggle becomes Allow across Search, Training and Agent. Either Block or Block on pages with ads becomes Search: Allow, Training: Disallow AI Training, and Agent: Block on pages with ads.

That last line adds something. Cloudflare's documentation describes the legacy toggle as blocking bots "classified as crawling for the purpose of AI training." The migrated configuration also restricts agents on ad-serving pages, a category the documentation does not list under the old switch. Neither the post nor the email presents this as an expansion.

For domains that had configured the granular controls, Search and Agent selections carry over unchanged, while Training selections of Block or Block on pages with ads both become Disallow AI Training. Cloudflare says this preserves "the practical effect" of earlier choices. One difference follows from its own description: the controls apply at domain level, and Disallow AI Training blocks every non-Accountable training crawler, so a site that had blocked training only on ad pages will now keep training-only crawlers off the whole domain. There is no ad-pages version of Disallow because, the post explains, the list of ad-serving pages "is too large and changes too frequently to enumerate in robots.txt."

Block AI Bots is deprecated in favour of the three granular controls, and Managed Robots.txt gives way to Bot Preference Sync. The email ties Cloudflare's AI bot blocking to Content Independence Day, which the company first marked in July 2025 with the private beta of pay per crawl, though the one-click switch now being retired is older: PPC Land reported Cloudflare's AI scraper block in June 2024.

Two presets for new domains

From September 15, a domain joining Cloudflare is offered one of two presets, depending on whether it earns money from advertising. Both enable Bot Preference Sync and allow Search. A site without ads also allows Training and Agent. A site with ads gets Training set to Disallow AI Training and Agent set to Block on pages with ads. Either can be changed during onboarding or later.

"Ad revenue depends on a human actually seeing the page. Training replaces that visit with an answer; agents fetch the page with nobody there to see the ads," the post states. Compared with the July defaults, training on new ad-supported domains moves from an ads-only block to a domain-wide robots.txt preference.

There is no Disallow option for agents. Agents, according to Cloudflare, do not create the same search trade-off, and no well-established directive exists for them yet; the company will revisit the question as standards such as ai-prefs mature. The IETF AI Preferences vocabulary draft reached version 07 on August 19, 2026, with several sections still lacking consensus.

The numbers Cloudflare puts forward

Two figures frame the demand: fewer than 1% of Cloudflare sites block Search bots, while 17% enable some mechanism to block training. On a network that sits in front of more than 20% of the web, that gap covers a great many domains.

On AI summaries, the post says more than half of consumers read them in search, that those consumers are more than 40% more likely to end a search after reading one, and that visitors referred by AI search convert at between three and more than five times the rate of those from traditional search. No source, sample or method is given for any of the three, so they stand as Cloudflare's claims rather than established measurements.

Independent research points the same way on clicks and scatters widely on conversion. Pew Research Center found in 2025 that Google users ended their browsing session on 26% of pages with an AI summary against 16% without, and a randomised study measured a 39.8% fall in clicks to publishers. Conversion estimates range from 23 times organic in Ahrefs' own data to roughly 3 times across publisher sites in Microsoft Clarity's analysis, which also cited an e-commerce study of 973 sites where ChatGPT referrals underperformed traditional channels.

The cost of the old trade-off is on record too: researchers at Rutgers and Wharton found that publishers blocking AI crawlers through robots.txt saw a 23.1% total traffic decline.

Declarations still depend on the reader

Cloudflare's earlier Content Signals directives were dismissed in July by Google's John Mueller, who said no crawler he knew of used them. This time the setting writes tokens the operators themselves document, such as Google-Extended and Applebot-Extended.

Compliance with robots.txt remains uneven. TollBit measurement found 15% of AI page fetchers in Europe reaching disallowed URLs. The mixed-use problem has also been shrinking at the margins: IAB Australia's guidance recorded mixed-purpose crawling falling from roughly 49% to 33% of AI requests in the first half of 2026 as operators split their bots. Among the largest operators, Apple, Google and Microsoft are the ones Cloudflare still treats as running combined crawlers.

Their reach is the reason verification matters. Cloudflare chief executive Matthew Prince argued in January that Googlebot reaches 3.2 times as many unique URLs as OpenAI's crawlers and 4.8 times as many as Microsoft's. Under Disallow AI Training, that crawler keeps fetching, and the main check on what happens next is URL-level reporting that Google says is weeks away.

Summaries are the next fight

"Mixed-use crawlers were the hard part of the training question. AI Summaries are next," the post states. A site-wide yes or no is, in Cloudflare's words, "too blunt," and the company aims by early next year to let site owners control how much of their content appears in summaries, set once on Cloudflare rather than operator by operator.

Prince said in July 2025 that Cloudflare would get Google to offer a way to block AI Overviews without blocking classic search indexing. A toggle came through regulation instead: Google began testing it on June 3, 2026, the day the UK Competition and Markets Authority imposed its first binding Publisher Conduct Requirement, whose substantive obligations take force on December 3, 2026.

Why this matters for publishers and advertisers

For ad-funded publishers, the update removes a risk the July design had created. A site with training blocks switched on would have lost Googlebot on September 15, and with it the search traffic that fills programmatic inventory. Under the new mapping it stays indexed. The cost has moved to Block, which now means leaving Google, Bing and Apple search altogether.

It does not settle the argument over AI answers. Whether a page feeds an AI Overview, the core of complaints such as Penske Media's claim that Google imposes a forced choice on publishers, still runs through Google's own domain-level toggle. For advertisers buying open-web inventory, that distinction decides how many human-viewed impressions survive the shift to summaries.

The measurement consequence may last longest. If the URL-level reporting promised under the third Accountable requirement arrives, publishers would for the first time see which pages were made available for training next to how those pages performed in search. Google's tools are described as weeks away and Apple's as due next year; Cloudflare Radar is where delivery will be tracked.

Timeline

Summary

Who: Cloudflare, through a post by Bryan Becker, Director, Product Security, and a follow-up email to customers. The change affects every Cloudflare customer that controls AI crawlers, the operators of the three mixed-use crawlers (Apple, Google and Microsoft), and the training crawlers run by Amazon, Anthropic, Meta and OpenAI.

What: A Disallow AI Training setting that writes a no-training preference into robots.txt, keeps Accountable mixed-use crawlers indexing for search and blocks all other training crawlers. Block and Block on pages with ads now apply to Googlebot, Applebot and Bingbot, search included. Block AI Bots and Managed Robots.txt are deprecated, existing settings migrate automatically, and new domains receive presets based on whether they carry ads.

When: The post is dated September 15, 2026, the date the changes took effect. A customer email dated September 16, 2026 said migration would run over the following week. Microsoft's robots.txt support and Cloudflare's AI summary controls are both targeted for early 2027.

Where: Across Cloudflare's network, configured per domain in Security Settings on all plans, with operator compliance tracked on Cloudflare Radar.

Why: Mixed-use crawlers tied search visibility to training access, and some 17% of Cloudflare sites block training while fewer than 1% block search. The July design would have cut sites blocking training off from Google, Apple and Microsoft search; the new setting separates the two, on the strength of operator commitments that are not yet fully delivered.