Google, Microsoft and OpenAI each said on October 30, 2026 that they will honor a single robots.txt instruction that lets a website refuse the use of its content for generative AI training across every participating provider at once, and the Anthropic engineer who wrote the specification told an Internet Engineering Task Force mailing list the same afternoon that his company had already started building support for it.

In Short

Four big AI companies have agreed to read the same short note on websites that says "do not use my pages to train AI." Until now, a website had to write a separate note for each company's robot, and each company used its own name for that note. The shared note is still a draft, and no company has said it reads it yet: Microsoft starts on February 1, 2027, Google plans to start in early 2027, and OpenAI and Anthropic have not given a date. Microsoft is also retiring its old opt-out method, and at Google the new note will also keep a site out of Gemini answers that cite web pages. Rankings in Google and Bing search results are not affected.

Four announcements, one mechanism

Google's post, "Supporting a shared AI training control for websites," appeared on the Google Search Central Blog on Friday, October 30, 2026, under the name of John Mueller, Google Search Advocate. Microsoft's Bing Blog published "Your content, your choice: clearer preferences for training and grounding," by Krishna Madhavan, Principal Product Manager, Microsoft Bing, on the same date. OpenAI posted "Making it easier for websites to express AI training preferences" in its Policies and Procedures section, credited to the company rather than to a named author. At 15:39 UTC, Kevin Kelley of Anthropic wrote to the ai-control mailing list of the IETF's AI Preferences working group, known as AIPREF, to say he had submitted a new Internet-Draft, draft-kelley-aipref-training-preference-signal-00.

All four describe the same device. A site operator writes an ordinary robots.txt group, but instead of naming a crawler on the User-Agent line, it names what the draft calls a "policy token." The draft's own illustration reads:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

According to the draft, that group "expresses a preference against AI training for all paths except those under /press-kit/, for which it expresses a preference in favor." Lookup follows the path matching and longest-match precedence of RFC 9309, the Robots Exclusion Protocol the IETF published in September 2022.

The rationale was put almost identically by each company. According to Google, website owners will no longer be "managing separate instructions for every AI developer." According to OpenAI, operators previously had to "identify individual crawlers and maintain separate rules for each and every provider." Microsoft described the aim as reducing "the complexity of navigating different platform preferences for the same use of content."

What the four did not share is a start date. Microsoft gave one: it will honor the new preferences from February 1, 2027, as it crawls. Google's blog post says "We aim to support this new control along with other AI providers in 2027," while a note Google added the same day to its crawler documentation is more specific, stating that Google "intends to support the AIPREF shared machine-readable training preference signal (with 'user-agent: __usage_ai-training') in early 2027." OpenAI said it aims "to implement the signal in the coming months." Kelley wrote that Anthropic "has already started implementing support for this draft," without saying when it would be live.

What the draft actually specifies

The document is short, ten pages including a long acknowledgments table, and its central claim is that it changes nothing about robots.txt itself. "It introduces no new directive names and no new grammar: the path language, longest-match precedence, and group structure of robots.txt are not changed," the abstract says. Kelley is the sole author, listed with an Anthropic affiliation. The draft's intended status is Standards Track, it was last updated on October 30, 2026, and it expires on May 3, 2027.

It lists five goals. Preferences can be bound to individual paths rather than only a whole site. Semantics stay as close as possible to RFC 9309. Expressing a preference should not require restating existing rules, because robots.txt files are size-constrained and "some deployed files already contain heavily duplicated groups." Parsers that do not implement the mechanism should behave exactly as before. And the mechanism "does not depend on the divergent parts of deployed parser behavior."

The token itself, __usage_ai-training, uses only the characters RFC 9309 allows in a product token: letters, underscores and hyphens. Consumers "MUST match policy tokens exactly, apart from case." The double underscore at the front has a practical purpose. According to the draft, "some deployed parsers match more flexibly; the __ prefix makes accidental matches unlikely." A single group may name several policy tokens, and policy tokens may sit in the same group as ordinary crawler names.

Kelley's message explained why the design leans so heavily on existing syntax. "I've spent a lot of time investigating how deployed parsers would interact with a wide variety of ways of expressing preferences in robots.txt files," he wrote. The mechanism, he added, "avoids relying on the areas where deployed parsers behave differently (and let me tell you, there are plenty!)."

The wildcard rule

One clause will matter more to most site owners than any other. "However, wildcard groups (i.e., User-Agent: *) MUST NOT be interpreted as matching the policy token," the draft states. If no group in a file names __usage_ai-training, "then no preference has been expressed."

The effect is that a site which currently blocks every bot with User-agent: * and Disallow: / has, under the draft, said nothing about AI training. Kelley described this as deliberate. "As a result, a robots.txt file that doesn't use this mechanism, as is true of every one published today, expresses no preference at all," he wrote. Rules addressed to crawler names, and to what the draft calls "proprietary policy tokens" such as Google-Extended, "keep whatever meaning they have today."

The draft also forbids the reverse inference. A consumer that finds only a proprietary token in a file should interpret it as it did before, but "MUST NOT treat it as expressing a preference in the context of this document." A Google-Extended block, in other words, does not count as a shared training opt-out for OpenAI or Microsoft.

Use, not access

The draft draws a hard line between what a crawler may fetch and what a company may do with the result. "This policy token is not a directive for crawling behavior," it says, and consumers "MUST interpret the RFC 9309 signals being expressed based on their own product token(s) exactly as they would in the absence of this document." Content-usage preferences and traffic management signals "are independent and orthogonal."

That is a direct answer to a gap Google's own staff described nine days earlier. In the second episode of its Non-Canonical Cabinet video series, Google Search Central said robots.txt limits AI crawling, not what bots do with data, calling RFC 9309 an access preference protocol. The policy token is an attempt to carry a statement about use inside the same file.

There is a small but telling gap between the draft and its author's description of it. Kelley wrote that "the draft does ask crawlers not to fetch what they can't use." The text itself is permissive. A consumer that crawls only for purposes covered by the token "MAY refrain from retrieving resources for which a preference against those purposes has been expressed." Under the IETF conventions the draft adopts, MAY marks an option, not a request. A site that wants a crawler to stay away still has to say so in a group for that crawler's own name.

Precedence, private deals and timing

Three further provisions shape how far the signal reaches. "Specific expressions SHOULD take precedence over general ones," the draft says, and consumers "MAY document that they interpret rules that match proprietary policy tokens as being more specific" than the shared token. Google has done exactly that, as described below.

The second provision concerns licensing. "Private agreements MAY override preferences expressed via this mechanism." A publisher with a content deal can therefore publish a blanket opt-out and still have that deal govern its relationship with a given AI company.

The third concerns time. The draft applies the AIPREF definitions to an "asset," defined as the content retrieved from a path, and states that "the preferences that apply to an asset are those expressed in the robots.txt file in effect when the asset was retrieved." On that wording, a preference attaches to content at the moment of collection. The draft does not address content gathered before a site adds the token.

What counts as training

The draft does not write its own definition. It takes one from draft-ietf-aipref-vocab-08, "A Vocabulary For Expressing AI Usage Preferences," dated September 13, 2026, by Paul Keller of Open Future and Martin Thomson of Mozilla. AI Training is "Using an asset to modify the learned parameters of a generative AI model."

The exclusions are as important as the definition. Training of models "used to perform exclusively non-generative tasks is not included in this category, even if the model is capable of generative tasks." And a generative AI model, under the vocabulary, "does not include a model that is used, or made available for use, for classification, ranking, or scoring, even where the model generates rationale for its output." Ranking systems of the kind that order search results or score ads fall outside the token on that reading.

Retrieval at answer time falls outside it too, because it does not change a model's parameters. Microsoft spelled this out by building a separate control for grounding, the practice of tying a generated answer to retrieved web pages. Google did not, and that difference produces the most consequential detail in the four announcements.

OpenAI's wording also departs from the vocabulary's. Its post says the signal concerns "the use of their content for foundational generative AI training." The vocabulary's definition covers any modification of a generative model's learned parameters, and OpenAI's post does not say whether "foundational" is meant to narrow that. OpenAI said it would publish details of its implementation later. Its existing training crawler, GPTBot, is described by PPC Land as OpenAI's training crawler for foundation models.

Google: a training signal mapped onto a broader token

Google's plan is a translation rule rather than a new system. "When a website uses this policy token to opt out of training in its robots.txt file (and Google-Extended is not explicitly specified), Google plans to interpret that as if the publisher had disallowed Google-Extended," the blog post says. Company-specific controls continue alongside, so a site can opt out broadly with the shared token while "tailoring their instructions to individual AI developers like Google through controls such as Google-Extended."

Google-Extended, which Google describes as "the AI training control we've offered since 2023," is not a crawler. According to Google's common crawlers page, it has no HTTP request user agent of its own; crawling is done with Google's existing user agent strings, and the token works only as a control. The same page says it lets publishers manage whether content Google crawls may be used for training future generations of the Gemini models behind Gemini Apps and the Vertex AI API for Gemini, and for grounding, which the page glosses as providing content from the Google Search index at prompt time "to improve factuality and relevancy," in Gemini Apps and in Grounding with Google Search on Vertex AI. It "does not impact a site's inclusion in Google Search," and the page indicates it is not used as a ranking signal.

Read together, the two documents produce an outcome the blog post does not state in so many words. The shared token is defined as a training preference only. Google-Extended covers training and grounding. Under Google's stated plan, a site that writes a disallow for __usage_ai-training and nothing for Google-Extended would be treated as having disallowed Google-Extended in full, which, according to Google's documentation, also withdraws its content from grounding in Gemini Apps and Vertex AI. The blog post approaches the same trade from the other side: "Websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps."

At Microsoft, the same robots.txt line refuses training and leaves grounding untouched. At Google, it refuses both. Google offers no separate grounding token, so a site that wants its pages cited in Gemini answers but not used for training has no single Google setting for that combination. This is PPC Land's reading of the published documentation; Google's post does not address the case.

The override works in one direction. Because Google treats Google-Extended as the more specific rule, a site can write a shared-token disallow and add an explicit Google-Extended allow group, keeping Gemini eligibility while opting out elsewhere. Google calls this the ability to "opt out across all providers, or to pick and choose." The blog post describes the site-wide case only. It does not say how path-level rules under the shared token will be translated, although Google-Extended itself accepts path rules, as the documentation's example group, with an allow for /archive/1Q84 and a disallow for /archive/, shows.

Nothing in the plan touches AI Overviews or AI Mode. Those features in Google Search are governed by the Googlebot rules and, since July 2026, by a Search Console setting: Google gave site owners a toggle to exit AI Overviews and AI Mode in July and extended it to every site by August 31, 2026. That control followed a UK requirement. When the Competition and Markets Authority imposed its conduct requirement on June 3, Google responded with a Search Console AI opt-out, and the CMA's page-level grounding controls come into force on March 3, 2027. In October 2025, publishers had pressed Google for a way to stay in search while leaving its AI features. For Google properties, a site now has three separate levers: Googlebot rules for crawling and Search, Google-Extended for Gemini training and grounding, and the Search Console control for AI features in Search. The shared token reaches only the second.

Google tied its decision to European law. "This proposal is consistent with the EU Code of Practice for General Purpose AI (GPAI) Model Providers under the EU AI Act that recognizes robots.txt as a primary method for web publishers to express machine-readable preferences," the post says. When the Commission published the final code in July 2025, its copyright chapter required signatories to use crawlers that read and follow robots.txt as specified in RFC 9309 and to identify other machine-readable protocols for rights reservations. A shared training token gives that obligation a concrete form across companies. Google also said the proposal "reflects feedback from publishers, civil society groups, regulators and other stakeholders that have called for industry-wide controls on AI training."

The documentation change was logged in Google's crawling changelog on October 30 under the heading "Updated common crawlers page to mention future AI training control support." It follows a run of crawler changes this year: the Google-Agent user agent on March 20, Web Bot Auth documentation on May 4, and on September 17 a rewrite of the Mediapartners-Google entry to clarify that it affects "various ad-related Google products (not just AdSense)," including Google Ad Manager. The changelog also records that on April 25, 2025, Google updated the Google-Extended description "Based on publisher feedback" to add "specificity and clarity."

Microsoft: training and grounding split, with deadlines

Microsoft's post is the longest and the only one with a transition plan. It opens with a statement of principle: "Publishers should not have to choose between being discoverable in Bing and deciding how generative models use their work." It covers web content "used for training or to inform generated responses, including web-grounded responses in Bing, Copilot, and Microsoft 365," and states that the preferences do not change eligibility for traditional Bing search results.

Microsoft claims priority. "Microsoft was the first company to provide publisher controls for AI training and grounding in 2023 with NoArchive and NoCache tags," the post says. Those tags are now being retired as AI controls. On the IETF process, Microsoft was candid that the work is unfinished: "Although the efforts of the IETF AI Preferences Working Group continue toward a final standard, we believe the current work represents significant progress that merits adoption."

Two preferences, three grounding settings

The first preference, AI Training, is the shared robots.txt token. Microsoft adds a page-level version that the IETF draft does not contain, set either in a meta tag or an HTTP header:

<meta name="Bingbot" content="ai-training:disallow">
X-Robots-Tag: Bingbot: ai-training:disallow

"The new AI Training preference set at a page level overrides the robots.txt setting," according to Microsoft. Because the page-level syntax addresses Bingbot by name, it is a Microsoft control, not part of the shared signal.

The second, AI Grounding, is Microsoft's alone and exists only at page level. "There is no site-wide robots.txt grounding preference," the post says. It has three values. Allow is the default. Disallow means content "is not used to help inform generated responses." The third, DisallowExtended, is new. It limits grounding to Microsoft-operated experiences, which the post lists as "AI Mode, AI Overview, Copilot, and Microsoft 365," and excludes "grounding services Microsoft provides to third parties, such as Web IQ and grounding through API access in Foundry."

That third setting reflects a business PPC Land has followed through 2026. Microsoft presented grounding as infrastructure for AI assistants in February, and in June introduced Web IQ, a set of grounding APIs sold to other companies' AI systems, initially in limited access to selected Azure customers. DisallowExtended lets a publisher stay in Copilot answers while keeping its pages out of the grounding Microsoft sells to others. The Bing search syndication API, which Microsoft says returns links for traditional search, is not affected by any grounding preference.

Both preferences default to Allow, and Microsoft attaches a warning to the restrictive options. Choosing DisallowExtended or Disallow for grounding will reduce brand visibility in AI experiences, the post says, adding that allowing grounding makes content "eligible to inform those responses, with links and citations providing a path to the original content." Then comes a sentence no other company published: "Neither choice guarantees inclusion, traffic, or payment." Microsoft also states that the preferences "do not establish licensing or payment arrangements." Publishers can already see part of that trade-off. Since February, Bing Webmaster Tools has shown how often AI systems cite a site's pages in Copilot and Bing's AI summaries.

The controls also stop short of access. "These settings govern use of content, not crawl access," the post says. "An Allow setting does not override a robots.txt rule blocking Bingbot from crawling." Nor does publishing a preference activate it: Microsoft has to crawl the page first. IndexNow can signal that a page has changed and raise the chance of a recrawl, according to Microsoft, "but it does not guarantee a recrawl date."

The transition calendar

Microsoft set four dates.

  • December 15, 2026: temporary grounding transition preferences open in Bing Webmaster Tools. The option is available only to sites using NoArchive or NoCache as of that date.
  • February 1, 2027: Microsoft starts honoring AI Training and AI Grounding preferences as pages are crawled.
  • March 1, 2027: after this date, "NoArchive and NoCache will no longer be read as a preference for training or grounding." Sites that use those tags and want to keep content out of grounding must, according to Microsoft, publish the new page-level grounding preference and record a temporary preference in Bing Webmaster Tools before then.
  • September 30, 2027: any temporary preference still in place expires. For each page, it applies "until recrawl or September 30, 2027, whichever is earlier."

The reason for the Webmaster Tools step is mechanical. Microsoft says it may not retrieve new preferences until it recrawls a site, "an activity that may not take place between February and March 1." There is no temporary setting for training. If a site using NoArchive or NoCache records no temporary grounding preference by March 1, its pages "default to Allow for grounding until Microsoft recrawls them."

One sentence in the post has a consequence that is easy to miss. "If we detect new AI Training or AI Grounding preferences on a site, we will disregard any NoArchive and NoCache preferences that may remain on that site." On that wording, a site that relies on NoCache to keep its pages out of Copilot answers, and then adds only the shared robots.txt training token, would lose that grounding restriction once Microsoft detects the new token, unless it also publishes page-level grounding preferences. Microsoft's post does not discuss that case directly.

OpenAI and Anthropic: commitments without dates

OpenAI's post runs to a few short paragraphs. "This voluntary effort makes it easier for publishers and website operators to communicate the same preference to multiple participating providers at once," it says. The company "will share another update with details about our implementation" when it implements the signal. Its existing controls stay: "This new signal will not replace the controls we already provide through robots.txt." The post links to OpenAI's crawler documentation and to the AIPREF vocabulary draft, and it ends with a request to competitors: "We encourage other AI providers to honor the signal."

OpenAI framed its participation as part of a group action. "We're joining other participants in the working group in today's announcement of this shared signal," the post says, adding that the company "will continue to adapt its practices as those conversations and the landscape continue to evolve in view of appropriate controls and fairness." The same day, OpenAI's site listed a separate company post titled "Fair use benefits everyone."

Anthropic's commitment came in Kelley's message to the working group. "I'm also excited to share that Anthropic has already started implementing support for this draft," he wrote. "This complements our existing crawler controls rather than replacing them." Those existing controls are three bots, which Anthropic described in February: ClaudeBot, which collects content that may contribute to training, Claude-User, which fetches pages when people ask Claude questions, and Claude-SearchBot, which supports search results.

Kelley was cautious about timing and scope. "It'll take us some time to add support across our services and work through the inevitable complications and edge cases that I'm sure we'll discover," he wrote. "Also, as standards in this area evolve, how we interpret and apply these signals may change." He presented early deployment as a source of evidence for the standards work, writing that he looked forward to "what we'll learn from having 'running code' in the field." He closed on the same note as OpenAI: "We hope that others will join us and make this a single signal that is respected across the Internet."

An individual draft inside a slower process

The signal four companies have committed to read has no official standing. The IETF's datatracker page states it plainly: "This I-D is not endorsed by the IETF and has no formal standing in the IETF standards process." It is an individual submission, not a working group document, and its stream state is listed as "No stream defined."

Kelley's message placed it inside the working group's unfinished business. He said the draft "differs from the attachment draft that we put aside in order to make progress on the vocabulary," and that he was "proposing this as input to the WG's attachment work," with a plan to open a pull request against that document. The attachment draft, "Associating AI Usage Preferences with Content in HTTP," is draft-ietf-aipref-attach-05, dated August 18, 2026, by Gary Illyes of Google and Martin Thomson. Illyes is also a co-author of RFC 9309. Google's post acknowledged the gap, saying the mechanism brings the training definition to site owners "while multistakeholder discussions continue on the rest of the AIPREF vocabulary." Microsoft referred to "the year plus effort" of the working group.

That sequencing matches what Illyes said in March 2025 about how robots.txt can evolve: no single company can change the protocol, and extensions need broad support first. Here the support arrived before the standard. Four large consumers have committed to an individual draft that expires in May 2027, ahead of any working group decision on how preferences attach to content.

The draft's acknowledgments show who took part in the discussion, without implying anyone endorsed the result. It thanks the working group chairs, Mark Nottingham of Cloudflare and Suresh Krishnan of Cisco, and the responsible Area Director, Mike Bishop of Akamai. Its table then lists 46 further contributors, noting that "Affiliations are listed for identification only, since IETF participants contribute as individuals." Four are from Google, including Illyes and Mueller, the author of Google's post. Four are from Microsoft, including Madhavan, the author of Microsoft's. OpenAI and Meta have two each. Anthropic's Daniel Etcovitch, Amazon Web Services' Elaine Newton and two Cloudflare staff appear. Publishing is represented by, among others, two people from News Corp and one each from the Financial Times, the BBC, Advance, Bertelsmann, Elsevier, Comcast NBCUniversal and the European Publishers Council. The list also includes the Electronic Frontier Foundation, the Center for Democracy and Technology, the Open Markets Institute, Creative Commons, the Internet Archive, two people from Common Crawl, and the data collection firms Bright Data and Sequentum.

Meta and Amazon are present in that table and absent from the commitments. Neither is named in the Google, Microsoft or OpenAI posts, or in Kelley's message, as a company that will read the token. Meta's position on crawling is in flux. PPC Land reported on October 17 that Meta's updated terms let it crawl advertisers' websites for AI, and on October 23 that Meta had delayed those terms to November 20 while saying content owners can use robots.txt to signal their preferences.

Why the plumbing matters to marketers

For publishers, the immediate gain is housekeeping. A training opt-out currently means one group per vendor token, and that list keeps growing. When Cloudflare added a Disallow AI Training option on September 15, its system wrote vendor tokens such as Google-Extended and Applebot-Extended into customers' robots.txt files. Apple, whose token is on that list, is not among the four companies that made commitments on October 30. Lists generated this way can also break: HasData found that a later Cloudflare change removed AI opt-outs from 118 sites' robots.txt files. Vendor-neutral tokens are not new either. In March, Czech publishers adopted a robots.txt standard with "MachineLearning" and "AI" user agents, but no AI company had committed to read them. What is new on October 30 is the commitment from the consumers.

The signal does not alter the basic shape of crawling. Googlebot and Bingbot are each a mixed-use crawler, collecting pages that serve search and AI products at once, and the draft leaves that arrangement in place: blocking a crawler still blocks search, while the policy token only states a preference about use. Nor does a stated preference guarantee behavior. TollBit data published in August found that about 15% of AI page fetchers in Europe reached URLs that sites had disallowed. The draft contains no enforcement mechanism, and OpenAI calls its own participation a "voluntary effort."

For advertisers and brands, the more consequential split is between training and grounding. Training shapes what future models know. Grounding decides which pages are cited, with links, in the answer a user reads now. Google ties the two together for Gemini; Microsoft separates them and warns that grounding restrictions reduce brand visibility in AI experiences. A retailer that wants its product pages cited in AI answers while keeping them out of model training faces different settings at each company, and at Google no single setting for that combination.

Several questions remain open. How will Google translate path-level shared-token rules into Google-Extended rules? Will OpenAI's "foundational" wording limit the opt-out to pre-training, or match the vocabulary's broader definition? Will Meta, Amazon or Apple follow? And what happens to the token if the working group's attachment draft takes a different form before Kelley's draft expires on May 3, 2027? Google and OpenAI have both promised further updates. Microsoft's calendar is the only one fixed so far, and its first date falls on December 15.

Timeline

Summary

Who: Google, Microsoft and OpenAI, through posts by John Mueller of Google, Krishna Madhavan of Microsoft Bing and OpenAI itself, and Anthropic, through Kevin Kelley, sole author of the IETF Internet-Draft draft-kelley-aipref-training-preference-signal-00. The change affects website operators, publishers and brands whose content is collected for generative AI.

What: The four companies committed to honor a shared robots.txt policy token, __usage_ai-training, which expresses a preference for or against using content to modify the parameters of generative AI models. Google will treat a shared-token opt-out as a Google-Extended disallow, which also covers grounding in Gemini Apps and Vertex AI. Microsoft adds a page-level training setting and a separate page-level grounding preference with a new DisallowExtended option, and retires NoArchive and NoCache as AI controls. Wildcard robots.txt groups do not count as a preference, and the token does not govern crawling.

When: All four statements are dated October 30, 2026. Microsoft starts honoring the preferences on February 1, 2027 and stops reading NoArchive and NoCache after March 1, 2027, with a Bing Webmaster Tools transition opening December 15, 2026. Google targets early 2027, OpenAI "the coming months," and Anthropic has started implementing without a date. The draft expires on May 3, 2027.

Where: In robots.txt files on websites worldwide, read by the participating companies' crawlers, and in discussion at the IETF's AI Preferences working group, where the draft has no formal standing.

Why: Site operators currently have to maintain separate rules for each AI provider. The companies say a single signal reduces that work and reflects calls from publishers, civil society groups and regulators for industry-wide training controls; Google also cites the EU's General-Purpose AI Code of Practice, which treats robots.txt as a primary way to express machine-readable preferences.