Google today published a technical guide for website operators who want its AI agent, Google-Agent, to get past their content delivery networks, web application firewalls and bot-detection services. The page, titled "Allow Google-Agent traffic" and dated November 17, 2026 in Google's crawling infrastructure documentation, tells security providers how to recognise the agent, says verified requests should skip bot challenges, and lists six reasons legitimate requests can still be refused after an allow rule is in place. Among them: per-IP rate limits that ignore the fact that many users' requests leave Google from the same pool of addresses.

In Short

Google has published instructions for letting its AI agent, which browses and acts on websites when a person asks it to, through the firewalls and bot filters that normally stop automated traffic. This matters to retailers, travel sites and publishers because the agent does more than read pages: Google's guide expects it to search, filter, add items to a cart and make bookings, and it generally ignores robots.txt rules. If you run a site, the practical change is that letting Google-Agent in now means checking a user-agent header, a published list of IP addresses and, for some requests, a cryptographic signature, then turning off CAPTCHAs and per-IP limits for traffic that passes.

What Google published today

The new page sits under Google's crawling infrastructure documentation on Google for Developers, next to the list of user-triggered fetchers where Google-Agent has been documented since March. Its opening line restates the agent's definition. "Google-Agent is the name used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request," according to the guide. The second sentence sets out the purpose: "Explicitly allowing Google-Agent traffic in your content delivery network (CDN), web application firewall (WAF), or bot-detection service helps prevent legitimate requests from being blocked while preserving your existing security controls."

The document has three parts: how Google-Agent identifies its requests, how to allow the traffic in a CDN, firewall or bot-detection service, and a troubleshooting list. It is short - three printed pages - and carries the footer "Last updated 2026-11-17 UTC."

Google flagged the addition this morning. According to Search Engine Roundtable, which reported the page at 7:05 am on November 17, Google wrote in its documentation changelog that it "Published a technical guide on how to ensure that Google-Agent has access to your website," adding that the purpose was "to help publishers that want to enable Google-Agent for their website." Barry Schwartz, who wrote the Search Engine Roundtable post, noted that Google-Agent itself is not new, and that the site had covered it in March.

What is new is the direction of the documentation. Until today, Google's pages on Google-Agent described what the agent is. The new guide describes what site operators and their security vendors would have to change for it to work.

How Google-Agent identifies itself

According to the guide, two signals identify every request. "Each Google-Agent request includes a User-Agent HTTP request header containing "Google-Agent" (see Google user-triggered fetchers) and comes from an IP address from user-triggered-agents.json."

The user agent strings are listed on Google's page of user-triggered fetchers, last updated on August 19, 2026. There are two variants.

Variant User agent string, as Google lists it
Mobile Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Google-Agent; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-agent)
Desktop Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Google-Agent; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-agent) Chrome/W.X.Y.Z Safari/537.36

Two details in those strings matter for anyone writing a matching rule. The Chrome version is a placeholder, W.X.Y.Z, so the number changes with the browser build and cannot be pinned. And the desktop string places the Google-Agent token inside the "KHTML, like Gecko" parentheses, before the Chrome version, rather than at the end of the string as the mobile variant does. A rule that looks for the token only in a trailing "compatible" block would miss desktop requests. The guide's own instruction avoids that problem by asking only whether the header "contains" Google-Agent.

The header alone proves little. Google's fetcher page carries a caution in bold: "The user agent string can be spoofed." That is why the guide pairs it with the address list. On the fetcher page, Google states that "The IP ranges the user-triggered fetchers use are published in the user-triggered-fetchers.json, user-triggered-fetchers-google.json, and user-triggered-agents.json objects." Google-Agent is tied to only one of them: "It uses IP ranges from user-triggered-agents.json."

The file is small. The version of user-triggered-agents.json served on November 17, which carries a creationTime of November 16, 2026 at 15:45, lists 20 address blocks: 12 IPv4 prefixes and 8 IPv6 prefixes. The first IPv4 entries listed fall in the 136.121 range, starting with 136.121.16.0/24, and the IPv6 entries begin at 2001:4860:c::/124. Google moved the refresh of its crawler IP range files from weekly to daily in March 2025, and moved the files to a new location in March 2026, so any copy kept in a firewall goes stale quickly. The guide says as much in its troubleshooting list.

IP range file What Google's fetcher page ties it to
user-triggered-fetchers.json User-triggered fetchers (general)
user-triggered-fetchers-google.json User-triggered fetchers (Google)
user-triggered-agents.json Google-Agent: "It uses IP ranges from user-triggered-agents.json"

The fetcher page also gives reverse DNS masks for the whole category. "The user-triggered fetchers' reverse DNS mask, depending on whether the fetcher is Google or user owned, matches ***-***-***-***.gae.googleusercontent.com or google-proxy-***-***-***-***.google.com respectively." The new guide does not mention reverse DNS at all. Its verification method rests on the header, the IP file and, where present, a signature.

The signature experiment

The third signal is optional and still experimental. "Google is also experimenting with the Web Bot Auth protocol using the https://agent.bot.goog identity," according to the guide. "In this experiment, Google signs requests made by Google-Agent with the HTTP Message Signatures standard (RFC 9421)."

Web Bot Auth is an IETF draft that lets an automated client sign its HTTP requests with a private key, so that a site can check the signature against a published public key instead of trusting an address or a header. The guide lists what signed requests carry.

Element Value or location, per Google
Signature-Agent header g="https://agent.bot.goog"
Signature and Signature-Input headers "labeled as g"
Signing standard HTTP Message Signatures, RFC 9421
Public key set https://agent.bot.goog/.well-known/http-message-signatures-directory

The guide calls these "Participating requests," and its allow rules say to verify the signature "If provided." Both phrases point to partial coverage. Google's separate Web Bot Auth page, last updated on October 13, 2026, is more direct: it says only a subset of Google-Agent requests is signed, and that sites should fall back to IP-based checks because not every request carries a signature. Google first described the experiment when it added Google-Agent to its list of user-triggered fetchers on March 20, 2026, and its changelog shows dedicated Web Bot Auth documentation arriving on May 4, 2026.

How the allow process works

The guide first points operators to their vendors. "Some bot-detection services, CDNs, and Firewalls provide ways to verify requests made by the Google-Agent user agent and allow these requests; refer to your provider for the exact steps," according to Google. No vendor is named.

For services that do not verify the agent automatically, the guide sets out four rules. According to Google, the sequence is:

  1. "Confirm the User-Agent header contains Google-Agent."
  2. "Confirm the IP address as being in user-triggered-agents.json."
  3. "If provided, confirm the signed request by verifying the Signature and Signature-Input headers for Signature-Agent: g="https://agent.bot.goog" according to Web Bot Auth."
  4. "For verified requests, skip bot challenges."

The fourth step is the one with commercial weight. A bot challenge - a CAPTCHA, a JavaScript test, an interstitial page - is the main tool sites use to stop automated traffic they cannot identify. Google's instruction is that traffic passing the first three checks should not see one at all.

Six reasons requests still fail

The troubleshooting section is the most revealing part of the document, because it describes what a working agent session looks like. It opens: "If legitimate Google-Agent requests are still blocked after allowing these requests:" and lists six checks.

Check Google's wording
Rule priority "Confirm that your allow rule has higher priority than (and evaluates before) broad cloud or data center ASN blocks, managed bot rules, or category-level AI agent block policies that could override a verified bot setting."
Interactive challenges "Confirm that your rule skips interactive CAPTCHAs and JavaScript challenges in addition to HTTP blocks, including standalone in-page CAPTCHA widgets triggered on search or form submission."
In-journey actions and APIs "Confirm that allowing these requests covers actions beyond initial homepage loads, including on-site search, filtering, clicking into detail pages, add-to-cart, booking, and underlying XHR, GraphQL, or api.* endpoints."
Shared-IP rate limits "Confirm that per-IP rate-limiting rules account for or exempt verified Google-Agent IP ranges, as concurrent requests from multiple users share the same egress pool."
IP range updates "Confirm that IP-based rules stay synced with the latest ranges in user-triggered-agents.json."
Proxy headers "Confirm that load balancers and intermediate proxies preserve the original client IP, the User-Agent header, and (for signed requests) the Signature, Signature-Input, and Signature-Agent headers."

Three of those checks say something about how the agent behaves.

It acts, not just reads

The in-journey item lists on-site search, filtering, product or listing detail pages, add-to-cart and booking, along with the XHR, GraphQL and api.* calls that sit underneath modern shop and travel front ends. That is a transaction path. Google's documentation does not name a product that drives these sessions, but the list makes plain that the agent is expected to move through a purchase or reservation flow, not stop at a homepage.

Many users, one address

The rate-limit item contains the clearest operational disclosure in the page: "concurrent requests from multiple users share the same egress pool." A site that throttles any single IP address after a set number of requests per minute would, in effect, be throttling every Google-Agent user routed through that address. Google does not say how many users share an address at any one moment. The pooling also means the IP address tells a site nothing about which person is behind a given request - the problem Cloudflare's November 5 position paper argued leaves websites carrying the cost of nearly all 14 agent-related problems.

Blanket AI blocks come first

The rule-priority item names three kinds of control that can override an allow rule: "broad cloud or data center ASN blocks, managed bot rules, or category-level AI agent block policies." The last of those is a product category several security vendors now sell - a switch that blocks AI agents as a class. Google's guidance is that an operator who wants Google-Agent in has to make its allow rule evaluate first, ahead of those defaults. Cloudflare, for its part, said in October that it would score verified bots on whether they respect site rules.

Robots.txt does not decide access

The guide says nothing about robots.txt. The omission follows from Google's classification. Google-Agent appears on the list of user-triggered fetchers, and that page's position on the file is explicit: "Because the fetch was requested by a user, these fetchers generally ignore robots.txt rules."

That leaves the CDN, the firewall and the bot-detection layer as the places where a site actually decides whether the agent gets in. Google's new page addresses only one side of that decision - how to let the agent through. It does not describe how to block it, and robots.txt, the tool most publishers use to manage Google's crawlers, generally does not reach it.

The position is consistent with what Google staff have said in public. On a podcast released on October 28, Google's Gary Illyes argued that fetchers acting for a single user have no good reason to obey robots.txt, and said faking Googlebot's user agent takes about 30 seconds. TollBit data published in August showed that about 15% of AI page fetchers in Europe reached URLs marked as disallowed.

The rest of Google's user-triggered fetcher list

Google-Agent is one of nine entries on the fetcher page, which Google describes as incomplete: "The list is not exhaustive, it only covers the requestors that are more likely to show up in log files and that we've received questions about."

Fetcher User agent token What it fetches, per Google
Chrome Web Store Google-CWS URLs developers provide in the metadata of Chrome extensions and themes
Feedfetcher FeedFetcher-Google RSS or Atom feeds for Google News and WebSub
Gemini Notebook Google-GeminiNotebook (former agent Google-NotebookLM, "supported until August 2026") Individual URLs that Gemini Notebook users provide as sources
Google-Agent Google-Agent Navigation and actions by agents hosted on Google infrastructure, on user request
Google Messages GoogleMessages Link previews for URLs sent in chat messages
Google Pinpoint Google-Pinpoint Individual URLs Pinpoint users specify as sources for document collections
Google Publisher Center GoogleProducer Feeds publishers supply for Google News landing pages
Google Read Aloud Google-Read-Aloud (former agent google-speakr, "deprecated") Pages read out using text-to-speech, on user request
Google Site Verifier Google-Site-Verification/1.0 Search Console verification tokens

Two of those entries show how the list has shifted this year. Google added the GoogleMessages fetcher on January 21, 2026, and its changelog records that the NotebookLM user agent became Google-GeminiNotebook on July 16, 2026, with the old value kept during a transition that the fetcher page says ran until August 2026.

Google-Agent is the only entry on the list that the documentation describes as performing actions. Every other fetcher retrieves a specific URL, feed or token. That difference is what the new guide is built around: a link preview needs one page, but a booking needs the whole site to cooperate.

What changed since March

Date Change Source
March 20, 2026 Google-Agent user agent and user-triggered-agents.json ranges added Google crawling changelog
May 4, 2026 Web Bot Auth documentation added Google crawling changelog
July 16, 2026 NotebookLM user agent changed to Google-GeminiNotebook Google crawling changelog
August 19, 2026 User-triggered fetchers page last updated Google fetcher page footer
October 13, 2026 Web Bot Auth page last updated, stating only a subset of Google-Agent requests is signed Google Web Bot Auth page
November 16, 2026 user-triggered-agents.json generated with 20 prefixes (12 IPv4, 8 IPv6) Google IP range file
November 17, 2026 "Allow Google-Agent traffic" guide published Google crawling documentation

When Google-Agent first appeared in March, Google gave Project Mariner, its experimental agent for completing tasks in a web browser, as the example product. Today's guide names no product at all.

Why this matters for marketers and publishers

For retailers, travel companies and anyone running paid traffic to a checkout, the guide is a reminder that defences built against scraping, card testing and credential stuffing judge automation largely by how it looks: a data centre address, scripted navigation, no human hand on the mouse. An agent sent to buy something can look much like one sent to steal something. Google is asking the operators who want its agent's business to carve out an exception, and to do it at several layers - rule order, challenge type, rate limits, header handling - at once.

How often those layers are in the way is already measurable. Blend's audit of 200 UK and US retailers found that bot protection blocked its AI shopping agent at 29 of them. Arkose Labs reported in October that only 2.3% of the AI agent sessions it classified as legitimate carried a valid Web Bot Auth signature, which explains why Google's guide still leans on IP files rather than signatures alone.

Google is not the largest source of agent traffic. HUMAN's October data, which PPC Land covered on November 13, put Meta's Muse at 59% of observed agent requests and showed a product it called Google Agent (Gemini), launched on October 8 according to HUMAN, at 0.8%. HUMAN did not tie that product to the Google-Agent user agent, and Google's documentation does not say which of its products send Google-Agent requests.

The competitive pattern is familiar. OpenAI's merchant playbook for ChatGPT Wallet, released on November 13, asks businesses to configure their CDN or firewall to recognise signed requests using Web Bot Auth. DataDome reported at the end of October that Meta had begun quietly signing some Muse requests. Each agent operator now publishes its own identity scheme and asks sites to put it on an allow list. Google's version comes with the added complication that the same company operates Googlebot, which sites cannot block without leaving Search, and a separate agent that sites can block only at the network edge.

For bot management teams, the shared egress pool is the detail likely to need the most work. Per-IP rate limiting is one of the oldest and cheapest controls on the web. Exempting a set of Google addresses from it means trusting Google's own traffic controls in its place, and the guide does not describe what those controls are.

What the guide leaves open

The page gives no volume figures, does not say what share of Google-Agent requests is signed, does not name the Google products that use the agent, and does not address reverse DNS or how a site would block the agent. The changelog entry Search Engine Roundtable quoted was not visible in the version of Google's crawling changelog PPC Land could load on November 17, which still showed October 30 as its latest update.

Key facts

  • Document: "Allow Google-Agent traffic," Google Crawling Infrastructure documentation, Google for Developers. Last updated 2026-11-17 UTC. URL: https://developers.google.com/crawling/docs/crawlers-fetchers/allow-google-agent
  • Related document: "List of Google user-triggered fetchers." Last updated 2026-08-19 UTC. URL: https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers
  • Agent name: Google-Agent (user-triggered fetcher; mobile and desktop user agent strings)
  • IP range file: user-triggered-agents.json; version generated 2026-11-16 15:45 lists 12 IPv4 and 8 IPv6 prefixes
  • Signature identity: https://agent.bot.goog; Signature-Agent header g="https://agent.bot.goog"; signing standard RFC 9421 (HTTP Message Signatures); key directory at /.well-known/http-message-signatures-directory
  • Allow rules: four (user agent, IP file, signature if provided, skip bot challenges for verified requests)
  • Troubleshooting checks: six (rule priority, interactive challenges, in-journey actions and APIs, shared-IP rate limits, IP range updates, proxy headers)
  • Robots.txt: user-triggered fetchers "generally ignore robots.txt rules," per Google's fetcher page
  • First report: Search Engine Roundtable, Barry Schwartz, November 17, 2026, 7:05 am

Timeline

Summary

Who: Google, which operates Google-Agent, and the website operators, CDNs, web application firewalls and bot-detection services that decide whether its requests get through. Search Engine Roundtable's Barry Schwartz first reported the page.

What: A new Google guide, "Allow Google-Agent traffic," explains how to identify the agent by its user agent header, its IP range file user-triggered-agents.json and, for some requests, a Web Bot Auth signature from https://agent.bot.goog. It sets four allow rules, ending with "For verified requests, skip bot challenges," and six troubleshooting checks covering rule priority, CAPTCHAs, add-to-cart and booking flows, shared-IP rate limits, IP updates and proxy headers.

When: The guide was published today, November 17, 2026. Google-Agent has been documented since March 20, 2026, and the IP file in use was generated on November 16.

Where: Google's crawling infrastructure documentation on Google for Developers, alongside the list of user-triggered fetchers. The rules apply to any site whose security layer sits in front of Google-Agent traffic.

Why: Google-Agent searches, filters, adds to cart and books on users' behalf, and security tools built to stop automation tend to stop it too. Because user-triggered fetchers generally ignore robots.txt, the network edge is where sites decide whether the agent gets in, and Google's guide covers only how to let it through.