Google has rewritten the robots.txt file at google.com so that 41 named AI crawlers, from OpenAI's GPTBot to ByteDance's Bytespider, receive a separate set of rules that shuts them out of every Google Maps page on the domain, and it has filed that group under __usage_ai-training, the shared AI training opt-out token that Google, Microsoft, OpenAI and Anthropic committed to honor on October 30, 2026.
The change was flagged today by Masahiro Tsuji, a Japanese SEO specialist, in a post on X. A copy of the file archived by the Internet Archive's Wayback Machine at 02:04 UTC on November 17 does not contain the new group, which places the edit somewhere in the five days before Tsuji's post. The file carries no date and no explanation. Besides the new group, Google rewrote an internal comment and added its own AI control, Google-Extended, to the file's default group, where it did not appear on November 17.
In Short
Google has added a block of rules to its own robots.txt file that names 41 AI crawlers, asks them to stay out of Google Maps pages, and tags the whole block with the new shared code for refusing AI training. Search results, Google's goto redirects and most other pages on the list were already off limits to every crawler, so the practical change is Maps plus a machine-readable statement that Google does not want those pages used to train AI models. Google's own AI training control sits in the default group instead, which keeps Maps pages open to it while closing them to the 41 others.
What Tsuji posted
Tsuji, who describes himself on X as an SEO specialist, published his post at 14:03 UTC on November 22. Translated from Japanese, it reads: "Google has made a large-scale update to google.com/robots.txt. It specifies many crawlers, mainly ones used for AI training, and blocks a great deal, including search results and goto-related pages. It has made its stance of refusing scraping clear in robots.txt as well." He linked the November 17 archived copy as the version before the update and wrote that everything from the __usage_ai-training line downward was new.
Two follow-ups narrowed the reading. In the first, Tsuji picked out the two Japanese names on the list, Cotoyogi and SBIntuitionsBot, and said he was surprised to see a crawler from Japanese academic institutions among them, pointing to the website of the Joint Support-Center for Data Science Research at Japan's Research Organization of Information and Systems (ROIS-DS). SB Intuitions, the SoftBank company that builds Japanese-language large language models, documents SBIntuitionsBot on its own crawler page.
In the second, he revised his headline claim. The rules covering search results and goto pages, he noted, appear to have sat under User-agent: * for a long time, and he asked whether the update simply makes that clearer by adding 42 user-agent specifications. The count matches the file: the new group opens with the policy token itself, followed by 41 crawler names.
Barry Schwartz, who runs Search Engine Roundtable, replied in the same thread: "technically, this was added back in July (was not there in June)." He linked an archived copy of the file dated July 24, 2026. The reply does not quote the line it means. It sits beneath Tsuji's remark about the search and goto rules, and the __usage_ai-training token itself was not published until October 30.
What changed between November 17 and November 22
The comparison below sets the archived November 17 copy against the version of the file that carried the new group on November 22.
| Element | November 17 copy | November 22 file |
|---|---|---|
User-agent: lines |
6 | 49 |
| Groups | 4 | 5 |
| Names in the default group | *, Yandex |
*, Google-Extended, Yandex |
__usage_ai-training group |
Absent | 42 User-agent: lines, 182 rules (154 Disallow, 28 Allow) |
| Maps rules for the 41 named crawlers | Default rules, including 37 Allow lines for Maps paths | Disallow: /maps |
| Comment above the Yandex group | 3 lines | 6 lines |
| Length | About 6,700 characters | About 12,500 characters (PPC Land count) |
The rest of the file is unchanged. The AdsBot-Google group, the Yandex group restricting careers search results, and the group that lets the link-preview crawlers facebookexternalhit and Twitterbot fetch /imgres and /search all read as before, and the file still ends with a pointer to https://www.google.com/sitemap.xml.
The comment Google rewrote
Comments in a robots.txt file are ignored by crawlers. They exist for the people who maintain it, which makes the rewritten one a rare window into how Google manages its own rules. The November 17 copy read:
# New user agent groups must also have a user agent reference in the global (*)
# group. See "Order of precedence" section in
# https://goo.gle/rep#order-of-precedence-for-user-agents
The November 22 file reads:
# Groups that name the same user agent are combined. A new user agent group
# must therefore also name its user agent in exactly one of:
# - the global (*) group, to inherit the default rules; or
# - the __usage_ai-training group below.
# Never list a user agent in both. See "Order of precedence" section in
# https://goo.gle/rep#order-of-precedence-for-user-agents
The logic comes from Google's published interpretation of the protocol. A crawler follows only the group with the most specific user agent that matches it, and ignores the rest, including the * group. Where several groups name the same crawler, Google's parser merges them. Yandex shows the technique: it is named in the default group and again in a short group of its own, so it receives the default rules plus two extra disallows for careers search results. The rewritten comment adds a second destination. A new named group must now be paired with either the default group or the AI training group, never both. A crawler named in both would have the two groups merged, and under longest-match precedence the default group's longer Maps allow rules, such as Allow: /maps/place/, would override the training group's single Disallow: /maps.
That design explains why the new group is so long. Naming the 41 crawlers in their own group cut them off from the default rules, so Google copied those rules into the group rather than relying on inheritance.
The 41 names
The group lists the token and then 41 user agents in alphabetical order:
| AI2Bot | Ai2Bot-Dolma | Amazonbot |
| Applebot-Extended | bedrockbot | Bytespider |
| CCBot | ChatGLM-Spider | ClaudeBot |
| cohere-training-data-crawler | Cotoyogi | DeepSeekBot |
| Diffbot | ElevenlabsCrawler | ExaSearchBot |
| FacebookBot | FriendlyCrawler | GPTBot |
| Grok | HySpider | ICC-Crawler |
| ImagesiftBot | img2dataset | KimiBot |
| LCC | Meta-ExternalAgent | MistralAI-Training |
| MistralBot | MoonshotBot | NagetBot |
| omgili | omgilibot | QwenBot |
| recraft-image-dataset | Reflectionbot | SBIntuitionsBot |
| Timpibot | VelenPublicWebCrawler | web-image-collection-research |
| Webzio-Extended | YiBot |
The roster reads like a census of model builders. It covers the large American developers through GPTBot, ClaudeBot, Amazonbot, Meta's FacebookBot and Meta-ExternalAgent, Cohere's training crawler and the Allen Institute for AI's two agents. It reaches European firms through Mistral's two entries, and names associated with Chinese model developers through Bytespider, DeepSeekBot, QwenBot, KimiBot, MoonshotBot, ChatGLM-Spider and YiBot. CCBot, the crawler behind the Common Crawl archive that many training datasets draw on, is there too. Not every entry is a crawler in the usual sense. Applebot-Extended is Apple's control token for AI training, the counterpart of Google's own, and img2dataset is an open-source tool for downloading image collections.
What the 41 crawlers actually lose
Before the change, none of the 41 names appeared in the file, so all of them followed the default group. That group already disallowed /search, /goto?, most of Google Books, Scholar, Patents, Shopping, Travel and dozens of legacy paths. The new group copies those rules almost line for line.
PPC Land compared the two rule sets entry by entry. Of the 235 rules in the default group, 181 appear unchanged in the AI training group. The 54 that do not, 37 Allow lines and 17 Disallow lines, all concern Google Maps, and the new group replaces them with a single line: Disallow: /maps. Under the protocol's prefix matching, that line covers every path that begins with those five characters.
| Path | Default group (*, Google-Extended, Yandex) |
__usage_ai-training group |
|---|---|---|
/maps/place/ |
Allowed | Disallowed |
/maps/search/ |
Allowed | Disallowed |
/maps/dir/ |
Allowed | Disallowed |
/maps/contrib/ |
Allowed | Disallowed |
/maps/d/ |
Allowed | Disallowed |
/maps/@ |
Allowed | Disallowed |
/maps/sitemap.xml, /maps/sitemaps/ |
Allowed | Disallowed |
/maps/reserve, /maps/reserve/partners |
Allowed | Disallowed |
/maps/about |
Allowed | Disallowed |
/search?*tbm=map |
Allowed | Disallowed |
/search |
Disallowed | Disallowed |
/goto? |
Disallowed | Disallowed |
/books/ |
Disallowed | Disallowed |
/scholar |
Disallowed | Disallowed |
So Tsuji's second reading is the accurate one. The 41 crawlers were asked to stay away from search results and goto redirects long before this edit. What is new for them is that Google Maps business listings, directions, contributor profiles, saved maps and the Maps sitemaps are now closed as well.
How the training token reads the file
The second effect does not depend on the crawler names at all. According to the draft specification behind the token, and to Google's October 30 statement, as PPC Land reported when the four companies announced their support, a company that implements the token reads google.com's file in five steps.
- It looks for a group that names
__usage_ai-training, matched exactly apart from case. Wildcard groups never count, and a file without the token "expresses no preference at all." Google's file now has such a group. - It applies the group's path rules as statements about use. The draft keeps the path syntax and longest-match precedence of RFC 9309, so the disallowed paths in Google's group, including
/search,/maps,/books/,/scholarand/shopping/product/, carry a preference against AI training. Explicitly allowed paths, such as/search/howsearchworks,/books/aboutand Scholar profile pages under/citations?user=, carry none. - It takes crawl permissions from its own product token, not from the policy token. "This policy token is not a directive for crawling behavior," the draft says. Bingbot, which is not named in Google's file, therefore keeps the default crawl rules, while Microsoft, which said it will begin honoring training preferences on February 1, 2027, would read a preference against training on content from those paths.
- It treats proprietary tokens as more specific. Google said it will read a shared-token opt-out as a Google-Extended disallow only when "Google-Extended is not explicitly specified."
- It attaches the preference at the moment of retrieval. The draft applies the robots.txt file "in effect when the asset was retrieved," and private agreements may override it.
Google has said it plans to support the token in 2027, and its crawler documentation says early 2027. Anthropic said on October 30 that it had already started implementing it. OpenAI gave no date.
Google's own control sits in the default group
Step four is where the new User-agent: Google-Extended line matters. Google-Extended has no crawler of its own. It is a control that decides whether content Google has already crawled may be used to train Gemini models and to ground answers in Gemini Apps and Vertex AI. By naming it in the default group, and following its own comment never to list a user agent in both groups, Google has explicitly specified it, so its October 30 translation rule does not apply to google.com.
Read against the table above, the result is an asymmetry. Under Google's file, Google-Extended follows the default rules, which leave /maps/place/, /maps/dir/ and the other Maps paths open. The 41 named crawlers are told to stay out of those paths, and every company that honors the shared token is asked not to train on them.
There is a fair reading in Google's favor. On November 17, Google-Extended was not named anywhere in the file, so it already fell under the default group, and the new line keeps it there once Google starts reading the token. Search results and goto pages remain disallowed for Google-Extended as well. Googlebot is not named either. As a mixed-use crawler, it gathers pages that serve both Search and Google's AI products, and it continues to follow the default rules. The file does not say why Google drew the line where it did.
Who is not on the list
The 41 names are training crawlers. Agents that fetch pages for search features or at a user's request are absent and continue to follow the default group: OpenAI's OAI-SearchBot and ChatGPT-User, Anthropic's Claude-SearchBot and Claude-User, PerplexityBot, Applebot, Bingbot and Googlebot. That matches the way Anthropic split its three crawlers in February, with ClaudeBot collecting material that may contribute to training and the other two serving search and user requests.
Meta appears on both sides. FacebookBot and Meta-ExternalAgent are in the AI training group, while facebookexternalhit, the agent that builds link previews when a URL is shared, keeps its own group permitting /imgres and /search. No SEO tool crawler, such as AhrefsBot or SemrushBot, is named in the new group.
Not served everywhere on November 22
The file Google returns does not appear to be uniform yet. Requests to https://www.google.com/robots.txt and https://google.com/robots.txt made by PPC Land between about 19:00 and 19:30 UTC today, including one with a query string appended, all returned the earlier version, without the training group, the Google-Extended line or the rewritten comment. Tsuji's post had shown the new version about five hours earlier. It is not clear whether Google is rolling the change out across its servers in stages or has partly reversed it.
A file that now matches the courtroom
Tsuji's "as well" points to everything else Google has done to make its results harder to collect. Google began requiring JavaScript for search in January 2025, and its later complaint against SerpApi said its SearchGuard system launched that month. On September 14, 2025, it dropped the num=100 parameter, multiplying the requests needed to collect 100 results.
On December 19, 2025, Google sued SerpApi under the anti-circumvention provisions of the Digital Millennium Copyright Act, a move PPC Land noted came from a company that is itself the web's largest crawler. The case has not gone Google's way so far. On November 9, 2026, Chief Judge Yvonne Gonzalez Rogers dismissed Google's amended complaint and gave the company 21 days to file again or see the case dismissed with prejudice, a deadline PPC Land counted as November 30.
The goto redirects in Tsuji's post belong to the same campaign. Result links routed through google.com/goto were first sighted on June 23, 2026, and Google confirmed the rollout on August 26, when Nozzle said resolving every link on a five-page ranking takes 500 to 1,000 requests. SISTRIX began resolving each wrapped link with an extra request, and SerpApi said on September 5 that it had restored direct URLs. If the line Schwartz meant is Disallow: /goto?, a July addition would fall between the first sighting and the confirmation. In October, Ahrefs said Google was poisoning 5% to 20% of the results served to suspected data providers. And on November 11, Brave said scrapers had moved to its own search engine after the goto change, at spikes of several thousand requests per second.
Against that record, a robots.txt line is the weakest instrument Google holds. In an interview published on October 28, Google's Gary Illyes called the file "a suggestion," and said faking Googlebot takes about 30 seconds. Google's own Search Central series said nine days before the token's launch that robots.txt limits AI crawling, not what bots do with data. The difference now is that google.com also states a preference about use, in the format four of the largest AI companies have promised to read.
Why it matters for marketers
For publishers weighing the new token, Google's file is a large implementation by one of the companies that backed it, and it is instructive. The draft lists, among its goals, that expressing a preference "should not require restating existing rules," because some deployed files "already contain heavily duplicated groups." Google chose to combine the token and the crawler names in one group, which the draft allows, and the price was restating 181 rules and roughly doubling the file. The draft also lets the token stand alone in a group, which states a training preference without changing any crawler's access.
For local businesses and the agencies that manage their listings, the Maps line is the change with practical reach. Business listings on place pages, directions and contributor profiles on google.com are now off limits to the 41 named crawlers, which narrows one public route by which AI training sets could collect Google's local data. It does nothing about crawlers that are not on the list or that ignore the file.
For SEO data providers, little changes. Search results were disallowed for every crawler before, and the providers Google is fighting do not identify themselves with any of the 41 names. The weight in that dispute lies with SearchGuard, the goto redirects and the courts.
What the file does not say
The file does not state when the edit was made, why Maps was singled out, why some well-known AI agents were left off, or why Google-Extended was placed in the default group. Google has not published a note alongside the change, and Schwartz's July reference remains unexplained.
Key facts
- File: robots.txt for www.google.com, served at
https://www.google.com/robots.txt. The file is undated. - Previous version: copy archived by the Wayback Machine at 02:04:21 UTC, November 17, 2026:
https://web.archive.org/web/20261117020421/https://www.google.com/robots.txt. - New group: opens with
User-agent: __usage_ai-training, followed by 41 crawler names; 182 rules (154 Disallow, 28 Allow);Disallow: /mapsreplaces 54 Maps-related default rules. - Default group:
User-agent: *,User-agent: Google-Extended,User-agent: Yandex; 235 rules (170 Disallow, 65 Allow). - User-agent lines in the file: 6 on November 17; 49 on November 22.
- Policy token:
__usage_ai-training, defined in draft-kelley-aipref-training-preference-signal-00, dated October 30, 2026, expiring May 3, 2027. - Google's rule for the token: a shared-token opt-out is read as a Google-Extended disallow only when "Google-Extended is not explicitly specified."
- Tsuji's post: Masahiro Tsuji (@tsuj) on X, 14:03 UTC, November 22, 2026,
https://x.com/tsuj/status/2124504163551961542; written in Japanese. - Japanese names Tsuji identified: Cotoyogi (ROIS-DS), SBIntuitionsBot (SB Intuitions).
- Related deadline: Google v. SerpApi, N.D. Cal. 4:25-cv-10826-YGR, order of November 9, 2026, 21 days to amend.
Timeline
- 2023 - Google introduces Google-Extended as its AI training control.
- January 2025 - Google starts requiring JavaScript for search.
- September 14, 2025 - Google drops the num=100 results parameter.
- December 19, 2025 - Google sues SerpApi over scraping of search results.
- February 2026 - Anthropic describes what ClaudeBot, Claude-User and Claude-SearchBot do.
- June 23, 2026 - Goto-style result links are first sighted.
- July 2026 - According to Barry Schwartz, a change discussed in Tsuji's thread is added to Google's robots.txt, citing an archived copy of July 24.
- August 26, 2026 - Google confirms the goto redirect rollout.
- September 5, 2026 - SerpApi restores direct URLs.
- October 21, 2026 - Google Search Central says robots.txt governs access, not data use.
- October 22, 2026 - Ahrefs says Google poisons 5% to 20% of results served to data providers.
- October 28, 2026 - Gary Illyes calls robots.txt "a suggestion".
- October 30, 2026 - Google, Microsoft, OpenAI and Anthropic commit to the
__usage_ai-trainingtoken. - November 9, 2026 - Court dismisses Google's amended complaint against SerpApi with 21 days to refile.
- November 11, 2026 - Brave says scrapers moved to Brave Search after Google's goto change.
- November 17, 2026, 02:04 UTC - The Wayback Machine archives google.com/robots.txt without the AI training group.
- November 22, 2026 - Masahiro Tsuji reports the new
__usage_ai-traininggroup in Google's robots.txt; some requests the same evening still return the earlier file. - November 30, 2026 - Google's deadline to file a further complaint against SerpApi.
- February 1, 2027 - Microsoft is scheduled to start honoring AI training preferences.
- May 3, 2027 - The token's draft specification is scheduled to expire.
Related PPC Land coverage
- Google, Microsoft, OpenAI and Anthropic back one AI training opt-out - the October 30 commitments to the token Google now uses on google.com, including Google's rule for Google-Extended.
- Google says robots.txt limits AI crawling, not what bots do with data - Google's own description of robots.txt as an access protocol.
- Faking Googlebot takes 30 seconds, Google's Gary Illyes says - Illyes on impostor crawlers and why the file is only a request.
- Anthropic clarifies what its three web crawlers do - and how to block them - the training, search and user-fetch split that Google's list mirrors.
- Google sues SerpApi over search scraping in copyright lawsuit - the December 2025 complaint and the SearchGuard system it describes.
- Google sues scraper while being the internet's biggest scraper itself - the tension between Google's crawling and its case against scrapers.
- Google faces 21-day deadline to save its DMCA case against SerpApi - the November 9 dismissal and the November 30 deadline.
- Google forces rank trackers into 500 to 1,000 requests per query - the goto redirects and their cost to rank tracking.
- Google's goto links force SISTRIX into added-cost tracking workaround - one vendor's response to the redirects.
- SerpApi restores direct URLs after Google goto redirect rollout - how quickly a scraping provider worked around goto.
- Ahrefs says Google poisons 5% to 20% of results served to SEO data tools - the altered results Google serves to suspected data providers.
- Google mandates JavaScript for search, impacting tools and accessibility - the January 2025 change that began the current anti-scraping cycle.
- Google eliminates n=100 SERP parameter forcing cost increases for SEO tools - the September 2025 removal of 100-result pages.
Summary
Who: Google, which maintains the robots.txt file for google.com; 41 named AI crawlers, including GPTBot, ClaudeBot, CCBot, Bytespider and the Japanese Cotoyogi and SBIntuitionsBot; Masahiro Tsuji, the Japanese SEO specialist who reported the change; and Barry Schwartz, who said part of it dates to July.
What: Google added a __usage_ai-training group to its robots.txt that names 41 AI crawlers, copies 181 of the default rules and replaces 54 Maps rules with Disallow: /maps. The only new crawl restriction for those crawlers is Google Maps, since search results and goto pages were already disallowed for all. Under the shared token, the group also states a preference against AI training on Google's search, Maps, Books, Scholar and other disallowed paths. Google named its own control, Google-Extended, in the default group, which leaves Maps paths open to it.
When: The change appeared between the Wayback Machine copy of November 17, 2026 and Tsuji's post today, November 22, 2026. Requests made the same evening still returned the earlier file. Microsoft plans to honor the token from February 1, 2027, and Google in 2027.
Where: At https://www.google.com/robots.txt, with the post published on X.
Why: The edit puts the token Google backed on October 30 to work on Google's own domain, adds a public statement against AI training to an anti-scraping campaign that runs from JavaScript requirements to a lawsuit against SerpApi, and shows how one of the token's sponsors chose to keep its own AI control outside the restrictions it sets for others.
Discussion