Google-Extended is a robots.txt product token that lets a website tell Google whether pages its crawlers collect may be used to train Gemini models and to ground answers in Gemini Apps and Vertex AI. It is not a crawler. No request ever arrives from Google-Extended, and no server log will record one. The token exists because Google fetches the web with shared infrastructure for search and for artificial intelligence (AI), and publishers wanted a way to refuse one use without vanishing from the other.

That separation is also the source of the controversy. The token leaves Google Search untouched in both directions: blocking it costs nothing in rankings, and it does nothing to keep content out of AI Overviews or AI Mode, the generative features built into Search.

How the token works

Google's crawler documentation lists Google-Extended alongside Googlebot, GoogleOther and its other common crawlers, with one caveat. Crawling is done with existing Google user agent strings, and the token is read "in a control capacity", according to the documentation. Googlebot fetches the page; robots.txt tells Google's downstream systems what they may do with the copy.

A site opts out entirely with a group containing the line "user-agent: Google-Extended" followed by "disallow: /". Rules can be scoped to paths. Google's own example allows a single archived page while disallowing the rest of its directory, so a publisher can offer its back catalogue to Gemini while withholding current reporting, or the reverse. Google's parser follows only the most specific group matching a user agent and generally caches robots.txt for up to 24 hours, so a change applies on the next refresh rather than instantly.

As of September 2026, the documentation defines two uses. The first is training future generations of Gemini models that power Gemini Apps and the Vertex AI API for Gemini, sold to developers through Google Cloud. The second is grounding, which Google describes as providing content from the Google Search index to the model at prompt time to improve factuality and relevancy, in Gemini Apps and in Grounding with Google Search on Vertex AI. Grounding is how a chatbot answer acquires live sources and citations. The token, the documentation adds, is neither a factor in inclusion in Google Search nor a ranking signal.

What it does not reach is set out elsewhere. Google's guidance on AI features describes AI as integral to how Search functions, and names robots.txt rules for Googlebot as the crawling control for Search. Limiting what appears in AI Overviews and AI Mode relies instead on nosnippet, data-nosnippet, max-snippet or noindex. Nor does an opt-out reach backwards. Google told the UK Competition and Markets Authority (CMA) that removing content already added to its search AI models was not feasible, and the regulator's June 2026 decision does not require it.

Origin and evolution

The token grew out of a consultation. On July 6, 2023, Danielle Romain, Google's vice president of trust, opened a public discussion on new publisher controls, writing that "existing web publisher controls were developed before new AI and research use cases". OpenAI supplied the first answer weeks later. Its GPTBot crawler, published in August 2023, gave sites a named training bot to block.

Google announced Google-Extended on September 28, 2023, again through Romain, as a way for publishers to decide whether their sites "help improve Bard and Vertex AI generative APIs, including future generations of models." The company promised to explore "additional machine-readable approaches to choice and control."

The wording has changed twice in substance. After Bard became Gemini in February 2024, the documentation switched to Gemini Apps and stated that the token did not affect inclusion or ranking in Search, according to Search Engine Journal. Google split its crawler documentation into per-crawler pages in September 2024 without changing behaviour. On April 25, 2025, it rewrote the description "to provide additional specificity and clarity" after publisher feedback, according to its crawling changelog. The new text named training explicitly, added grounding with a definition, and stated that the token is not used as a ranking signal.

Why it matters

For publishers whose audiences depend on search, Google-Extended is the only standing Google control that separates AI use from search visibility, and uptake has trailed its rivals. By the end of 2023, 24% of the most widely used news websites across ten countries blocked Google's AI crawler, against 48% for OpenAI, according to the Reuters Institute. Rates ranged from 60% in Germany to 7% in Poland and Spain, and 97% of sites blocking Google also blocked OpenAI.

Wider samples show the same gap. Originality.ai counted 13.6% of the top 1,000 websites blocking the token in 2024, against 35.7% for GPTBot. BuzzStream, a digital PR software vendor, found 46% of 100 leading UK and US news sites blocking it in December 2025, compared with 62% for GPTBot and 75% for CCBot; US publishers stood at 58%, UK publishers at 29%.

Reluctance has a commercial logic. BuzzStream later found that 92.3% of top news sites blocking Google-Extended still appeared in AI citations across ChatGPT, Gemini, AI Overviews and AI Mode. Hangcheng Zhao of Rutgers and Ron Berman of Wharton found that news publishers blocking large language model crawlers, a set including Google-Extended, lost about 7% of weekly visits within six weeks. A Cloudflare Radar snapshot dated August 31, 2026 found the token in 572 disallow rules and 267 allow rules, according to an analysis by TechnologyChecker.

Industry bodies now treat such tokens as a category of their own. Guidance from IAB Australia in July 2026 separates control tokens from crawlers, noting that they never visit a site and cover model training only.

Limitations and disputes

Scope is the principal complaint. Publishers can refuse Gemini while their content keeps feeding the AI inside Search, which reaches far larger audiences. Testimony at the Department of Justice search remedies hearing sharpened the point. On May 2, 2025, Eli Collins, a Google DeepMind vice president, agreed that the search organisation could train on data publishers had opted out of, "for use in search", according to Nieman Lab. An internal document dated August 26, 2024 recorded that filtering out opted-out material removed 80 billion of 160 billion tokens, according to Bloomberg.

Challenges followed. The Independent Publishers Alliance, Movement for an Open Web and Foxglove complained to the European Commission on June 30, 2025. Penske Media, owner of Rolling Stone and Variety, sued on September 12, 2025, alleging that Google trains search-specific products including AI Overviews on content opted out through Google-Extended. The Commission opened its own investigation on December 9, 2025. Google's answers leaned on the existing token. "There's an overall opt-out on the training side," Robby Stein, Google's vice president of product for Search, said on October 10, 2025, describing the controls open to publishers.

Bundling is contested from the other side. One rule covers both training and grounding, so a site refusing training also forgoes Gemini citations. Google told the CMA that separate controls would invite free-riding, with publishers opting in to grounding for traffic while opting out of the training that makes the products possible. The CMA declined to require further separation of the token outside Search. Cloudflare argued in January 2026 that Google-Extended and nosnippet had failed publishers and that Googlebot should be split into separate crawlers. Alternatives have not taken: Google's John Mueller said in July 2026 that he knew of no crawler using Cloudflare's content-signal directives.

Verification is the last gap. With no user agent of its own, the token cannot be audited from logs. A site sees Googlebot either way and must take the downstream separation on trust.

Not the same as

Googlebot. The crawler that actually fetches. Blocking it removes a site from Search and, with it, from Google's AI products; Explaining Googlebot covers its mechanics.

The Search Console generative AI control. A domain-level setting that keeps pages out of AI Overviews, AI Mode and generative features in Discover. It does not stop training, which remains the token's job.

Google-CloudVertexBot. A genuine crawler that fetches pages when a site owner asks Google to build Vertex AI Agents from them, with no effect on Search.

Applebot-Extended. Apple's control token, introduced in June 2024 with the same no-crawl design; Apple rewrote its documentation in June 2026 to disclose foundation model training. GPTBot, by contrast, is a real crawler that can be refused at the network edge.

Recent developments

Regulation is now reshaping the token's reach. The CMA designated Google with strategic market status in general search on October 10, 2025 and imposed a Publisher Conduct Requirement on June 3, 2026. Google confirmed during the process that it could ensure content withheld through Google-Extended would not be added back into fine-tuning models used only in search generative AI features, and later said it did not anticipate building an additional control. The decision brings training, including fine-tuning, of those features within scope for outputs shown to UK users. Main obligations apply from December 3, 2026; page-level grounding controls follow on March 3, 2027.

Google has meanwhile built a Search-side counterpart. A Search Console toggle tested with UK sites from June 3 was documented on July 20 and extended to every site on August 31, 2026. Sulina Connal, Google's managing director for news and books partnerships in EMEA, had called the work a "huge engineering project" in February. Training, however, still runs through Google-Extended alone.

Timeline

  • July 6, 2023: Google opens a public discussion on AI-era web publisher controls
  • August 2023: OpenAI publishes GPTBot, the first widely blocked AI training crawler
  • September 28, 2023: Google announces Google-Extended for Bard and Vertex AI generative APIs
  • February 2024: Documentation updated for Gemini Apps after the Bard rename
  • February 22, 2024: Reuters Institute reports 24% of top news sites in ten countries blocking Google's AI crawler at the end of 2023
  • June 2024: Apple introduces Applebot-Extended
  • August 2024: Originality.ai finds 13.6% of the top 1,000 websites blocking Google-Extended
  • August 26, 2024: Internal Google document records 80 billion of 160 billion tokens removed by publisher opt-outs
  • September 16, 2024: Google splits its crawler documentation into per-crawler pages
  • April 25, 2025: Google-Extended description rewritten to name training and grounding
  • May 2, 2025: Eli Collins testifies that the search organisation can train on opted-out content for use in search
  • June 30, 2025: Independent publishers file an EU complaint over AI Overviews
  • September 12, 2025: Penske Media sues Google over AI use of publisher content
  • October 10, 2025: CMA designates Google with strategic market status; Robby Stein cites the training opt-out
  • December 9, 2025: European Commission opens an investigation into Google's AI content practices
  • December 18, 2025: BuzzStream finds 46% of 100 top UK and US news sites blocking the token
  • January 30, 2026: Cloudflare calls for Googlebot to be split into separate crawlers
  • February 11, 2026: Sulina Connal describes AI opt-out controls as a huge engineering project
  • March 2026: BuzzStream finds 92.3% of news sites blocking Google-Extended still cited in AI answers
  • June 3, 2026: CMA imposes the Publisher Conduct Requirement; Google begins testing a Search Console opt-out
  • July 20, 2026: Google documents the Search Console generative AI control
  • July 28, 2026: IAB Australia releases guidance classifying control tokens separately from crawlers
  • August 31, 2026: The control reaches all websites; a Cloudflare Radar snapshot records 572 disallow rules naming the token
  • December 3, 2026: Main CMA obligations take effect
  • March 3, 2027: Page-level grounding controls due under the CMA requirement

Summary

Who. Google operates the token and the Gemini, Vertex AI and Search products it governs. Publishers and site owners set the rules in robots.txt, while regulators including the UK Competition and Markets Authority and the European Commission, and litigants such as Penske Media, contest how far it reaches.

What. A robots.txt product token with no user agent string of its own. It controls whether content Google crawls may train future Gemini models and ground answers in Gemini Apps and Vertex AI, without affecting Search inclusion or ranking, and without covering AI Overviews or AI Mode.

When. Announced on September 28, 2023 for Bard and Vertex AI, redefined on April 25, 2025 to name training and grounding, and brought under UK regulatory obligations from June 3, 2026, with main duties applying from December 3, 2026.

Where. In the robots.txt file at the root of any domain, read by Google's existing crawlers worldwide, with the CMA's added obligations applying to outputs shown to UK users.

Why. Google fetches the web once for search and AI alike, so a token was the least disruptive way to offer a choice over AI use. Its narrow scope, which leaves the AI inside Search untouched, is why publishers, Cloudflare and regulators continue to press for more.