A mixed-use crawler is a single automated program, identified by one user agent name, that collects web pages for more than one purpose at once. In the case that matters most, the fetch that places a page in a search engine's index also supplies text for training or grounding artificial intelligence (AI) models. The term exists because the web's main access control, the robots.txt file, works by crawler name rather than by purpose. When one name covers both jobs, a site owner can refuse the AI use only by refusing search as well.

Cloudflare, whose definitions have done most to fix the term, describes it as "a single crawler doing both Search and Training" in a post published on September 15, 2026. Its earlier wording, from July, also swept in agents: bots that blend search, agent use, and training behind a single user agent. Mixed-purpose, dual-purpose and multi-purpose are common synonyms. Googlebot, Bingbot and Applebot are the standard examples.

One fetch, several products

The mechanics start with the Robots Exclusion Protocol, a 1994 convention formalised as RFC 9309 in September 2022. A robots.txt file lists user agent tokens and the paths each may fetch. It governs access, not what happens to a page afterwards.

Operators have handled that gap in two ways. OpenAI and Anthropic run separate crawlers: GPTBot for training and OAI-SearchBot for search, or ClaudeBot, Claude-SearchBot and Claude-User, each with its own token. A site can disallow the training crawler and leave the search crawler alone.

Google, Apple and Microsoft chose the other route. Their search crawlers fetch once, and control tokens or page tags tell the operator how the result may be reused. Google-Extended, introduced on September 28, 2023, governs whether crawled content may train Gemini models and ground answers in Gemini Apps and Vertex AI. Google's documentation states that it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal". No bot called Google-Extended exists; crawling uses existing Google user agent strings, and the token acts only as a control.

Apple's Applebot-Extended, documented on June 11, 2024, follows the same design. "Applebot-Extended does not crawl webpages," according to Apple's support page, which says Applebot data may also help train Apple foundation models. Microsoft relies on meta tags instead. Since September 22, 2023, content marked NOARCHIVE has been excluded from Bing's chat answers and from training Microsoft's generative foundation models, while still appearing in search results, according to the Bing Webmaster blog.

Cost explains the preference. Googlebot is one client of a shared fetching platform, Google engineers said in March 2026, and that platform also serves Gemini, Shopping and AdSense. Fetching once is cheaper than crawling the web twice.

From search engine to answer engine

Search crawlers always did more than index, building snippets, caches and previews. The label acquired weight once generative AI turned the same corpus into answers that compete with the pages they draw on. Bing Chat's launch on February 7, 2023, GPTBot's arrival that August and AI Overviews in the United States in May 2024 set the pattern: independent AI firms got named, blockable crawlers, while incumbents folded AI into existing ones.

The limits of Google's control became public in May 2025. Asked by Department of Justice lawyer Diana Aguilar whether the search organisation could train on data publishers had opted out of, DeepMind vice president Eli Collins replied: "Correct - for use in search," according to Bloomberg's reporting as summarised by Nieman Lab. Google-Extended never reached AI Overviews.

Regulators followed. The EU's General-Purpose AI Code of Practice, finalised on July 10, 2025, encourages signatories that also run search engines to ensure a training opt-out "does not directly lead to adverse effects on the indexing" of that content, according to its copyright chapter. The European Commission opened an antitrust investigation on December 9, 2025 into whether publishers could refuse AI use without losing Google Search.

Cloudflare took up the infrastructure side. Its Content Signals Policy, released on September 24, 2025, added search, ai-input and ai-train preferences to robots.txt files on more than 3.8 million domains using its managed service, according to Cloudflare. On January 30, 2026, it urged the UK Competition and Markets Authority (CMA) to force a split of Googlebot, publishing data showing Googlebot reached 1.70 times more unique URLs than ClaudeBot and 1.76 times more than GPTBot.

Scale and measurement

Estimates of how much traffic mixed-use crawlers generate depend on the denominator. Cloudflare put them at 36.6% of verified crawler traffic on its network in September 2026, the largest single category, according to its press release. IAB Australia's guidance, drawing on first-half 2026 figures, recorded mixed-purpose crawling falling from roughly 49% to 33% of AI requests as operators split composite bots into narrower tokens. The two measure different populations and do not form a trend.

Behaviour towards these crawlers is lopsided. Fewer than 1% of Cloudflare sites block search crawlers, while 17% restrict AI training, according to Cloudflare. Between July 2025 and January 2026, sites using its blocking tool refused GPTBot and ClaudeBot nearly seven times as often as Googlebot and Bingbot.

Why publishers and advertisers care

For ad-funded sites the issue is arithmetic. A search listing can deliver a reader who sees an advertisement; an AI answer built from the same fetch may replace that visit. Cloudflare's July 1 post made the same point: an ad signals that a site owner "meant for a person to land there and see it".

Blocking has a price. News publishers that refused AI crawlers through robots.txt lost roughly 7% of weekly traffic within six weeks in research from Rutgers and Wharton. Yet the most common blocks spare mixed-use crawlers altogether, since refusing Googlebot means leaving Google Search. That structure has pushed some publishers to discuss delisting from Google entirely. Retailers face the trade from the other side: 62.5% of training crawler activity targets retail sites, so a blocking rule also decides whether a catalogue appears inside AI shopping answers.

Separation, verification and who decides

The central dispute is whether mixed-use crawlers should exist at all. The BBC, Guardian Media Group, Cloudflare and others told the CMA that enhanced controls were no substitute for separate crawlers. Google called separation an extreme structural remedy, while Microsoft, which runs its own mixed-use crawler, and the Chamber of Progress called it disproportionate, according to the CMA's final decision of June 3, 2026.

The regulator found the two approaches "comparably effective" but chose enhanced controls as more proportionate. Citing third-party estimates, it noted that Googlebot accounted for about 50% of web crawls in May 2025, far more than any other Google crawler. Its consultation put third-party costs of separation at 25 million to 50 million pounds a year, and Google claimed at least 150 million pounds for itself, according to OpenAttribution's published response, which questioned both.

Controls carry limits of their own. The CMA requirement obliges Google to offer opt-outs covering training and grounding, yet Google told the regulator it could not remove content already used to fine-tune its search AI models before an opt-out. The Search Console control documented on July 20, 2026 works at domain level and does not stop model training. Google had earlier called a separate AI Overviews opt-out a "huge engineering project".

Verification is the weakest point. A robots.txt preference cannot prove what an operator did with a page, and Google's John Mueller said on July 6, 2026 that Content Signals directives have no effect on any crawler or language model. Classification is disputed too. Cloudflare counts Meta among operators that separate search from training, yet Meta's documentation assigns its Meta-ExternalAgent crawler both model training and direct indexing, alongside the newer Meta-WebIndexer. And the arbiter is a private company: Cloudflare sits in front of more than 20% of websites, and its rules bind nothing beyond them.

Not the same as

Control token. Google-Extended and Applebot-Extended never fetch anything. They are instructions that ride on a mixed-use crawler's visits.

User-triggered fetcher. ChatGPT-User, Claude-User and Google-Agent retrieve pages because a person asked. Cloudflare classes them as Agent traffic, outside its September definition.

Undeclared crawler. A stealth crawler hides its identity. A mixed-use crawler announces its name but not which use a given fetch will serve.

Recent developments

The largest shift came on September 15, 2026. Cloudflare had announced in July that from that date multi-purpose crawlers would be judged by their most restrictive behaviour, so sites blocking training would also block Googlebot, Applebot and Bingbot. In August it attached four disclosure conditions for mixed-use operators seeking access to sites that refuse training. On the day, the plan softened. A new Disallow AI Training setting writes a no-training preference into robots.txt, keeps crawlers designated Accountable available for search and blocks every other training crawler, including the training-only bots of Amazon, Anthropic, Meta and OpenAI. Existing training blocks migrated to it automatically. The Block setting now stops Googlebot, Applebot and Bingbot entirely, search included.

Apple, Google and Microsoft received the designation on the strength of current controls and dated commitments, according to Cloudflare. Google plans URL-level transparency tools for Google-Extended within weeks, Apple expects a URL-level tool next year, and Microsoft targets robots.txt support for a no-training preference in early 2027. Until then, the setting conveys nothing to Bing through robots.txt.

Revision 08 of the Internet Engineering Task Force (IETF) AI Preferences vocabulary, posted on September 14, 2026, keeps train-ai and search as distinct categories. The CMA's main obligations take effect on December 3, 2026, with page-level grounding controls due by March 3, 2027.

Timeline

  • 1994: Martijn Koster proposes the robots exclusion convention
  • September 2022: The convention is published as IETF RFC 9309
  • February 7, 2023: Microsoft launches the new Bing with chat answers built on its search index
  • August 2023: OpenAI publishes GPTBot as a dedicated training crawler
  • September 22, 2023: Bing introduces NOCACHE and NOARCHIVE controls for chat answers and model training
  • September 28, 2023: Google launches the Google-Extended control token
  • May 2024: AI Overviews launch in the United States
  • June 11, 2024: Apple documents Applebot-Extended
  • July 2024: OpenAI introduces OAI-SearchBot alongside SearchGPT
  • May 2025: DeepMind's Eli Collins testifies that the search organisation can train on opted-out data for use in search
  • July 10, 2025: The EU General-Purpose AI Code of Practice is finalised
  • September 24, 2025: Cloudflare releases its Content Signals Policy
  • October 10, 2025: The CMA designates Google with strategic market status in general search
  • December 9, 2025: The European Commission opens an antitrust investigation into Google's use of publisher content for AI
  • January 30, 2026: Cloudflare calls on the CMA to require separate Google crawlers
  • February 11, 2026: A Google executive describes an AI Overviews opt-out as a major engineering project
  • June 3, 2026: The CMA imposes its Publisher Conduct Requirement and Google begins testing a Search Console opt-out
  • July 1, 2026: Cloudflare introduces Search, Agent and Training categories and a most-restrictive rule for multi-purpose crawlers
  • July 6, 2026: John Mueller says Content Signals directives have no effect
  • July 20, 2026: Google documents the Search Console generative AI control
  • August 21, 2026: Cloudflare publishes Bot Preference Sync and four conditions for mixed-use operators
  • September 14, 2026: Revision 08 of the IETF AI Preferences vocabulary is posted
  • September 15, 2026: Cloudflare launches Disallow AI Training and designates Apple, Google and Microsoft as Accountable
  • December 3, 2026: Main CMA publisher obligations take effect
  • Early 2027: Microsoft's target date for honouring a robots.txt no-training preference
  • March 3, 2027: Page-level grounding controls due under the CMA requirement

Summary

Who: Operators of search crawlers that also feed AI, chiefly Google, Microsoft and Apple, and the publishers, retailers and advertisers whose pages they fetch. Cloudflare, the UK Competition and Markets Authority, the European Commission and the IETF shape the rules around them.

What: A single crawler, under one user agent name, whose fetches serve both a search index and AI training or grounding. Because robots.txt controls access by name, separate control tokens such as Google-Extended and Applebot-Extended, or meta tags such as Bing's NOARCHIVE, are needed to govern reuse.

When: The problem took shape between February 2023 and May 2024 as Bing Chat and AI Overviews built answers on search indexes. Regulatory and infrastructure responses followed from July 2025, culminating in the CMA requirement of June 3, 2026 and Cloudflare's Disallow AI Training setting of September 15, 2026.

Where: Across the open web, with the most detailed rules in the United Kingdom, where the CMA's obligations apply from December 3, 2026, and on the more than 20% of websites behind Cloudflare.

Why: Refusing a mixed-use crawler means leaving a search engine, so publishers cannot withhold content from AI without sacrificing referral traffic. That bundling is now contested through regulation, network-level enforcement and standards work, with crawler separation still pressed by publishers and resisted by the largest operators.