Google has rewritten the robots.txt file at google.com so that 41 named AI crawlers, from OpenAI's GPTBot to ByteDance's Bytespider, receive a separate set of rules that shuts them out of every Google Maps page on the domain, and it has filed that group under __usage_ai-training, the shared AI training opt-out token that Google, Microsoft, OpenAI and Anthropic committed to honor on October 30, 2026.

The change was flagged today by Masahiro Tsuji, a Japanese SEO specialist, in a post on X. A copy of the file archived by the Internet Archive's Wayback Machine at 02:04 UTC on November 17 does not contain the new group, which places the edit somewhere in the five days before Tsuji's post. The file carries no date and no explanation. Besides the new group, Google rewrote an internal comment and added its own AI control, Google-Extended, to the file's default group, where it did not appear on November 17.

In Short

Google has added a block of rules to its own robots.txt file that names 41 AI crawlers, asks them to stay out of Google Maps pages, and tags the whole block with the new shared code for refusing AI training. Search results, Google's goto redirects and most other pages on the list were already off limits to every crawler, so the practical change is Maps plus a machine-readable statement that Google does not want those pages used to train AI models. Google's own AI training control sits in the default group instead, which keeps Maps pages open to it while closing them to the 41 others.

What Tsuji posted

Tsuji, who describes himself on X as an SEO specialist, published his post at 14:03 UTC on November 22. Translated from Japanese, it reads: "Google has made a large-scale update to google.com/robots.txt. It specifies many crawlers, mainly ones used for AI training, and blocks a great deal, including search results and goto-related pages. It has made its stance of refusing scraping clear in robots.txt as well." He linked the November 17 archived copy as the version before the update and wrote that everything from the __usage_ai-training line downward was new.

Two follow-ups narrowed the reading. In the first, Tsuji picked out the two Japanese names on the list, Cotoyogi and SBIntuitionsBot, and said he was surprised to see a crawler from Japanese academic institutions among them, pointing to the website of the Joint Support-Center for Data Science Research at Japan's Research Organization of Information and Systems (ROIS-DS). SB Intuitions, the SoftBank company that builds Japanese-language large language models, documents SBIntuitionsBot on its own crawler page.

In the second, he revised his headline claim. The rules covering search results and goto pages, he noted, appear to have sat under User-agent: * for a long time, and he asked whether the update simply makes that clearer by adding 42 user-agent specifications. The count matches the file: the new group opens with the policy token itself, followed by 41 crawler names.

Barry Schwartz, who runs Search Engine Roundtable, replied in the same thread: "technically, this was added back in July (was not there in June)." He linked an archived copy of the file dated July 24, 2026. The reply does not quote the line it means. It sits beneath Tsuji's remark about the search and goto rules, and the __usage_ai-training token itself was not published until October 30.

What changed between November 17 and November 22

The comparison below sets the archived November 17 copy against the version of the file that carried the new group on November 22.

Element November 17 copy November 22 file
User-agent: lines 6 49
Groups 4 5
Names in the default group *, Yandex *, Google-Extended, Yandex
__usage_ai-training group Absent 42 User-agent: lines, 182 rules (154 Disallow, 28 Allow)
Maps rules for the 41 named crawlers Default rules, including 37 Allow lines for Maps paths Disallow: /maps
Comment above the Yandex group 3 lines 6 lines
Length About 6,700 characters About 12,500 characters (PPC Land count)

The rest of the file is unchanged. The AdsBot-Google group, the Yandex group restricting careers search results, and the group that lets the link-preview crawlers facebookexternalhit and Twitterbot fetch /imgres and /search all read as before, and the file still ends with a pointer to https://www.google.com/sitemap.xml.

The comment Google rewrote

Comments in a robots.txt file are ignored by crawlers. They exist for the people who maintain it, which makes the rewritten one a rare window into how Google manages its own rules. The November 17 copy read:

# New user agent groups must also have a user agent reference in the global (*)
# group. See "Order of precedence" section in
# https://goo.gle/rep#order-of-precedence-for-user-agents

The November 22 file reads:

# Groups that name the same user agent are combined. A new user agent group
# must therefore also name its user agent in exactly one of:
# - the global (*) group, to inherit the default rules; or
# - the __usage_ai-training group below.
# Never list a user agent in both. See "Order of precedence" section in
# https://goo.gle/rep#order-of-precedence-for-user-agents

The logic comes from Google's published interpretation of the protocol. A crawler follows only the group with the most specific user agent that matches it, and ignores the rest, including the * group. Where several groups name the same crawler, Google's parser merges them. Yandex shows the technique: it is named in the default group and again in a short group of its own, so it receives the default rules plus two extra disallows for careers search results. The rewritten comment adds a second destination. A new named group must now be paired with either the default group or the AI training group, never both. A crawler named in both would have the two groups merged, and under longest-match precedence the default group's longer Maps allow rules, such as Allow: /maps/place/, would override the training group's single Disallow: /maps.

That design explains why the new group is so long. Naming the 41 crawlers in their own group cut them off from the default rules, so Google copied those rules into the group rather than relying on inheritance.

The 41 names

The group lists the token and then 41 user agents in alphabetical order:

AI2Bot Ai2Bot-Dolma Amazonbot
Applebot-Extended bedrockbot Bytespider
CCBot ChatGLM-Spider ClaudeBot
cohere-training-data-crawler Cotoyogi DeepSeekBot
Diffbot ElevenlabsCrawler ExaSearchBot
FacebookBot FriendlyCrawler GPTBot
Grok HySpider ICC-Crawler
ImagesiftBot img2dataset KimiBot
LCC Meta-ExternalAgent MistralAI-Training
MistralBot MoonshotBot NagetBot
omgili omgilibot QwenBot
recraft-image-dataset Reflectionbot SBIntuitionsBot
Timpibot VelenPublicWebCrawler web-image-collection-research
Webzio-Extended YiBot

The roster reads like a census of model builders. It covers the large American developers through GPTBot, ClaudeBot, Amazonbot, Meta's FacebookBot and Meta-ExternalAgent, Cohere's training crawler and the Allen Institute for AI's two agents. It reaches European firms through Mistral's two entries, and names associated with Chinese model developers through Bytespider, DeepSeekBot, QwenBot, KimiBot, MoonshotBot, ChatGLM-Spider and YiBot. CCBot, the crawler behind the Common Crawl archive that many training datasets draw on, is there too. Not every entry is a crawler in the usual sense. Applebot-Extended is Apple's control token for AI training, the counterpart of Google's own, and img2dataset is an open-source tool for downloading image collections.

What the 41 crawlers actually lose

Before the change, none of the 41 names appeared in the file, so all of them followed the default group. That group already disallowed /search, /goto?, most of Google Books, Scholar, Patents, Shopping, Travel and dozens of legacy paths. The new group copies those rules almost line for line.

PPC Land compared the two rule sets entry by entry. Of the 235 rules in the default group, 181 appear unchanged in the AI training group. The 54 that do not, 37 Allow lines and 17 Disallow lines, all concern Google Maps, and the new group replaces them with a single line: Disallow: /maps. Under the protocol's prefix matching, that line covers every path that begins with those five characters.

Path Default group (*, Google-Extended, Yandex) __usage_ai-training group
/maps/place/ Allowed Disallowed
/maps/search/ Allowed Disallowed
/maps/dir/ Allowed Disallowed
/maps/contrib/ Allowed Disallowed
/maps/d/ Allowed Disallowed
/maps/@ Allowed Disallowed
/maps/sitemap.xml, /maps/sitemaps/ Allowed Disallowed
/maps/reserve, /maps/reserve/partners Allowed Disallowed
/maps/about Allowed Disallowed
/search?*tbm=map Allowed Disallowed
/search Disallowed Disallowed
/goto? Disallowed Disallowed
/books/ Disallowed Disallowed
/scholar Disallowed Disallowed

So Tsuji's second reading is the accurate one. The 41 crawlers were asked to stay away from search results and goto redirects long before this edit. What is new for them is that Google Maps business listings, directions, contributor profiles, saved maps and the Maps sitemaps are now closed as well.

How the training token reads the file

The second effect does not depend on the crawler names at all. According to the draft specification behind the token, and to Google's October 30 statement, as PPC Land reported when the four companies announced their support, a company that implements the token reads google.com's file in five steps.

  1. It looks for a group that names __usage_ai-training, matched exactly apart from case. Wildcard groups never count, and a file without the token "expresses no preference at all." Google's file now has such a group.
  2. It applies the group's path rules as statements about use. The draft keeps the path syntax and longest-match precedence of RFC 9309, so the disallowed paths in Google's group, including /search, /maps, /books/, /scholar and /shopping/product/, carry a preference against AI training. Explicitly allowed paths, such as /search/howsearchworks, /books/about and Scholar profile pages under /citations?user=, carry none.
  3. It takes crawl permissions from its own product token, not from the policy token. "This policy token is not a directive for crawling behavior," the draft says. Bingbot, which is not named in Google's file, therefore keeps the default crawl rules, while Microsoft, which said it will begin honoring training preferences on February 1, 2027, would read a preference against training on content from those paths.
  4. It treats proprietary tokens as more specific. Google said it will read a shared-token opt-out as a Google-Extended disallow only when "Google-Extended is not explicitly specified."
  5. It attaches the preference at the moment of retrieval. The draft applies the robots.txt file "in effect when the asset was retrieved," and private agreements may override it.

Google has said it plans to support the token in 2027, and its crawler documentation says early 2027. Anthropic said on October 30 that it had already started implementing it. OpenAI gave no date.

Google's own control sits in the default group

Step four is where the new User-agent: Google-Extended line matters. Google-Extended has no crawler of its own. It is a control that decides whether content Google has already crawled may be used to train Gemini models and to ground answers in Gemini Apps and Vertex AI. By naming it in the default group, and following its own comment never to list a user agent in both groups, Google has explicitly specified it, so its October 30 translation rule does not apply to google.com.

Read against the table above, the result is an asymmetry. Under Google's file, Google-Extended follows the default rules, which leave /maps/place/, /maps/dir/ and the other Maps paths open. The 41 named crawlers are told to stay out of those paths, and every company that honors the shared token is asked not to train on them.

There is a fair reading in Google's favor. On November 17, Google-Extended was not named anywhere in the file, so it already fell under the default group, and the new line keeps it there once Google starts reading the token. Search results and goto pages remain disallowed for Google-Extended as well. Googlebot is not named either. As a mixed-use crawler, it gathers pages that serve both Search and Google's AI products, and it continues to follow the default rules. The file does not say why Google drew the line where it did.

Who is not on the list

The 41 names are training crawlers. Agents that fetch pages for search features or at a user's request are absent and continue to follow the default group: OpenAI's OAI-SearchBot and ChatGPT-User, Anthropic's Claude-SearchBot and Claude-User, PerplexityBot, Applebot, Bingbot and Googlebot. That matches the way Anthropic split its three crawlers in February, with ClaudeBot collecting material that may contribute to training and the other two serving search and user requests.

Meta appears on both sides. FacebookBot and Meta-ExternalAgent are in the AI training group, while facebookexternalhit, the agent that builds link previews when a URL is shared, keeps its own group permitting /imgres and /search. No SEO tool crawler, such as AhrefsBot or SemrushBot, is named in the new group.

Not served everywhere on November 22

The file Google returns does not appear to be uniform yet. Requests to https://www.google.com/robots.txt and https://google.com/robots.txt made by PPC Land between about 19:00 and 19:30 UTC today, including one with a query string appended, all returned the earlier version, without the training group, the Google-Extended line or the rewritten comment. Tsuji's post had shown the new version about five hours earlier. It is not clear whether Google is rolling the change out across its servers in stages or has partly reversed it.

A file that now matches the courtroom

Tsuji's "as well" points to everything else Google has done to make its results harder to collect. Google began requiring JavaScript for search in January 2025, and its later complaint against SerpApi said its SearchGuard system launched that month. On September 14, 2025, it dropped the num=100 parameter, multiplying the requests needed to collect 100 results.

On December 19, 2025, Google sued SerpApi under the anti-circumvention provisions of the Digital Millennium Copyright Act, a move PPC Land noted came from a company that is itself the web's largest crawler. The case has not gone Google's way so far. On November 9, 2026, Chief Judge Yvonne Gonzalez Rogers dismissed Google's amended complaint and gave the company 21 days to file again or see the case dismissed with prejudice, a deadline PPC Land counted as November 30.

The goto redirects in Tsuji's post belong to the same campaign. Result links routed through google.com/goto were first sighted on June 23, 2026, and Google confirmed the rollout on August 26, when Nozzle said resolving every link on a five-page ranking takes 500 to 1,000 requests. SISTRIX began resolving each wrapped link with an extra request, and SerpApi said on September 5 that it had restored direct URLs. If the line Schwartz meant is Disallow: /goto?, a July addition would fall between the first sighting and the confirmation. In October, Ahrefs said Google was poisoning 5% to 20% of the results served to suspected data providers. And on November 11, Brave said scrapers had moved to its own search engine after the goto change, at spikes of several thousand requests per second.

Against that record, a robots.txt line is the weakest instrument Google holds. In an interview published on October 28, Google's Gary Illyes called the file "a suggestion," and said faking Googlebot takes about 30 seconds. Google's own Search Central series said nine days before the token's launch that robots.txt limits AI crawling, not what bots do with data. The difference now is that google.com also states a preference about use, in the format four of the largest AI companies have promised to read.

Why it matters for marketers

For publishers weighing the new token, Google's file is a large implementation by one of the companies that backed it, and it is instructive. The draft lists, among its goals, that expressing a preference "should not require restating existing rules," because some deployed files "already contain heavily duplicated groups." Google chose to combine the token and the crawler names in one group, which the draft allows, and the price was restating 181 rules and roughly doubling the file. The draft also lets the token stand alone in a group, which states a training preference without changing any crawler's access.

For local businesses and the agencies that manage their listings, the Maps line is the change with practical reach. Business listings on place pages, directions and contributor profiles on google.com are now off limits to the 41 named crawlers, which narrows one public route by which AI training sets could collect Google's local data. It does nothing about crawlers that are not on the list or that ignore the file.

For SEO data providers, little changes. Search results were disallowed for every crawler before, and the providers Google is fighting do not identify themselves with any of the 41 names. The weight in that dispute lies with SearchGuard, the goto redirects and the courts.

What the file does not say

The file does not state when the edit was made, why Maps was singled out, why some well-known AI agents were left off, or why Google-Extended was placed in the default group. Google has not published a note alongside the change, and Schwartz's July reference remains unexplained.

Key facts

  • File: robots.txt for www.google.com, served at https://www.google.com/robots.txt. The file is undated.
  • Previous version: copy archived by the Wayback Machine at 02:04:21 UTC, November 17, 2026: https://web.archive.org/web/20261117020421/https://www.google.com/robots.txt.
  • New group: opens with User-agent: __usage_ai-training, followed by 41 crawler names; 182 rules (154 Disallow, 28 Allow); Disallow: /maps replaces 54 Maps-related default rules.
  • Default group: User-agent: *, User-agent: Google-Extended, User-agent: Yandex; 235 rules (170 Disallow, 65 Allow).
  • User-agent lines in the file: 6 on November 17; 49 on November 22.
  • Policy token: __usage_ai-training, defined in draft-kelley-aipref-training-preference-signal-00, dated October 30, 2026, expiring May 3, 2027.
  • Google's rule for the token: a shared-token opt-out is read as a Google-Extended disallow only when "Google-Extended is not explicitly specified."
  • Tsuji's post: Masahiro Tsuji (@tsuj) on X, 14:03 UTC, November 22, 2026, https://x.com/tsuj/status/2124504163551961542; written in Japanese.
  • Japanese names Tsuji identified: Cotoyogi (ROIS-DS), SBIntuitionsBot (SB Intuitions).
  • Related deadline: Google v. SerpApi, N.D. Cal. 4:25-cv-10826-YGR, order of November 9, 2026, 21 days to amend.

Timeline

Summary

Who: Google, which maintains the robots.txt file for google.com; 41 named AI crawlers, including GPTBot, ClaudeBot, CCBot, Bytespider and the Japanese Cotoyogi and SBIntuitionsBot; Masahiro Tsuji, the Japanese SEO specialist who reported the change; and Barry Schwartz, who said part of it dates to July.

What: Google added a __usage_ai-training group to its robots.txt that names 41 AI crawlers, copies 181 of the default rules and replaces 54 Maps rules with Disallow: /maps. The only new crawl restriction for those crawlers is Google Maps, since search results and goto pages were already disallowed for all. Under the shared token, the group also states a preference against AI training on Google's search, Maps, Books, Scholar and other disallowed paths. Google named its own control, Google-Extended, in the default group, which leaves Maps paths open to it.

When: The change appeared between the Wayback Machine copy of November 17, 2026 and Tsuji's post today, November 22, 2026. Requests made the same evening still returned the earlier file. Microsoft plans to honor the token from February 1, 2027, and Google in 2027.

Where: At https://www.google.com/robots.txt, with the post published on X.

Why: The edit puts the token Google backed on October 30 to work on Google's own domain, adds a public statement against AI training to an anti-scraping campaign that runs from JavaScript requirements to a lawsuit against SerpApi, and shows how one of the token's sponsors chose to keep its own AI control outside the restrictions it sets for others.