Gary Illyes of Google said it would take him "probably 30 seconds" to set up a script that crawls the web looking like Googlebot, in a podcast interview published on October 28, 2026, in which he described catching impostors with honeypot websites, called the web's main crawler-control file "a suggestion" and explained why Google has started signing some requests from its AI agents with cryptographic keys.

In Short

Any program that visits a website can claim to be Google's crawler just by writing Google's name on its request, and a Google employee who spent years on its crawler team says setting that up takes about 30 seconds. That matters because many sites wave Google's crawler through, while a fake one might copy content for unknown purposes or hit a site so hard that real visitors can't load it. The only reliable check today is whether the visit comes from internet addresses Google publishes, and Google is testing digital signatures on some requests that a forger cannot copy.

The interview

The conversation appeared on the YouTube channel Odys Podcast: The High Stakes Growth Show under the title "Anyone Can Fake Googlebot in 30 Seconds | Gary Illyes, Google," dated October 28, 2026. Host Aneta Samkoff introduced it as recorded at Shenzhen SEO, the conference in the Chinese city. The video runs to about 33 and a half minutes, including an opening sponsor segment. On October 31, the YouTube page showed 684 views and seven comments on a channel with 3,460 subscribers and 82 episodes. The channel describes itself as hosting "conversations with founders, operators, investors, affiliates, marketers and SEOs scaling businesses across competitive verticals."

According to the episode description, Illyes "spent years on the crawler team and on web standards work," including robots.txt and the early stages of Web Bot Auth, and "also runs his own honeypot sites to watch who is crawling the web." Illyes, of Google Search Relations, said he and the host chose crawling because it is a "very hot topic on the internet nowadays," adding that "there are a few myths that are worth debunking or at least explaining in more detail."

The interview landed in a busy month for him. On October 2, at Search Central Live Deep Dive Europe in Barcelona, he showed timing data indicating that new pages take about 20 hours to be found and some are never discovered. On October 21, he co-presented a Google Search Central video in which Google said robots.txt limits AI crawling, not what bots do with data.

Quotations below come from YouTube's automatic transcript. Names the captions garble, such as Googlebot, robots.txt, IETF and Web Bot Auth, are written in their correct form; other caption errors are listed near the end of this article. Everything Illyes said is his account, not independently measured findings.

Three layers of identity

The first question was simple: if a request claims to be Googlebot, how does a site know it is Google? Illyes started with the weakest signal. "All Google crawlers identify themselves in some ways. The most prominent way and also the most spoofable way is the user agent header," he said.

Every HTTP client sends that header with each request, naming the engine behind it. For Google, he said, "it would be something like Mozilla blah blah blah version blah blah blah and it also has the name typically Googlebot or Mediapartners-Google or whatever crawler it is." The header, "as I said is very easy to spoof."

Two other layers exist. One is the reverse DNS entry attached to the requesting IP address: running the host command on it should return a hostname "that points to something like Google but some numbers dot googlebot.com." The other is the address itself, which Illyes called "the most valuable right now." The reason is publication. "We publish also Apple and Bing, we all publish our crawling IP ranges and then based on the IP you can identify that this request is coming from Bing or it is coming from Google or it's coming from Apple and so on. OpenAI does the same thing."

Google's documentation adds a step he did not mention. Its page "Verify requests from Google crawlers and fetchers," last updated on March 20, 2026, describes a reverse lookup, a check that the hostname belongs to googlebot.com, google.com or googleusercontent.com, and then a forward lookup on that hostname that must return the original IP address. The forward check is what stops the operator of a foreign address block from publishing a reverse record that simply claims a googlebot.com name.

The same page lists hostname masks by crawler type: crawl-***-***-***-***.googlebot.com for common crawlers such as Googlebot, rate-limited-proxy-***-***-***-***.google.com for special-case crawlers, and ***-***-***-***.gae.googleusercontent.com or google-proxy-***-***-***-***.google.com for user-triggered fetchers. The address ranges sit in five JSON files, from common-crawlers.json to user-triggered-agents.json. In March 2025, Illyes announced on LinkedIn that Google had moved the refresh of those files from weekly to daily, and a year later Google moved them to a new home under developers.google.com/crawling/ipranges/.

Thirty seconds and one line of code

Can anyone simply put Googlebot in the user agent? "Yeah and that is the problem," Illyes said. "Changing anyone's user agent header is super simple. Like really really simple." In Chrome's inspection tool, "with I think like three two clicks I can change my user agent header to Googlebot."

Scripts are quicker still. "If you are doing programmatic crawling like you set up a script and then just crawl the internet they're setting the user agent header is literally one line of code and it can be anything like there's no reservation on what can you do and what you cannot. So basically if I wanted to crawl as Googlebot it would take me probably 30 seconds to set up a script."

How common is it? "Unfortunately very," he said. There are "various efforts on the internet" to reduce it, "but everything on the internet takes very long time usually." It "definitely is something that is happening always have been happening," and "something that I think nowadays we have to do something about."

Honeypots and unknown data centres

His evidence is personal. "I personally know because I set up honey traps on the internet," Illyes said. The sites carry "some unique and useful content" of the kind crawlers want, and he waits to see who arrives. Years on Google's crawler team mean he recognises Google's own addresses, so the mismatch stands out in his logs: "the user agent header was something something Googlebot but then the IP I have no idea where it's coming from. Like I can't identify some data center in whatever country."

He gave no count, no share of traffic, no number of sites and no time period. The judgment that impersonation is "very" common rests on his own observation, not on published data.

Security firms have published numbers on related forgery. GreyNoise said in August 2026 that scanners on 824 addresses forged the names of 13 AI crawlers while hunting credential files, and none of those addresses matched the published IP lists of Anthropic, OpenAI, Google, Perplexity or Amazon. The same research counted 263,849 forged sessions using the name Google-Extended - a token that, according to Google's documentation, has no HTTP user agent string of its own, so no genuine request ever carries it.

What an impostor can do

The visit is not the risk, in Illyes's telling. "Well, expectations," he said. With Googlebot, "you kind of know what's going to happen with the crawled pages like Googlebot is feeding search." With an impersonator, "you simply just don't know what is going to happen with your content." Google made the same point in a March 31 blog post, which said that Googlebot activity in server logs reflects Google Search and nothing else - a statement that only holds for requests that pass verification.

The second concern is load. "We spent 10 years fine-tuning our crawler infrastructure to ensure that we are not breaking the internet," Illyes said. "If you are not careful especially with these new data centers that have terabit connections and whatnot it's super easy to overwhelm a site." A spoofer offers no assurance at all: "will it crawl like 10,000 times the same URL in two seconds or it's just going to fetch the page once and then goes away."

He has framed Google's own system in those terms before. In a Search Off the Record episode in March, he described Googlebot as one client of an internal crawling platform whose task is to fetch pages without breaking the internet.

What is the worst thing an unknown bot can do? "Bring it down." Every site has limited bandwidth - "I don't know like 10 megabit connection or something," he offered. A script looping through a list of crawled URLs every 200 milliseconds would eventually exhaust it, "and then real visitors cannot go to your site and I think that's the worst." He recalled running an image hosting site around 2004 or 2005 that drew what he called a "massive" denial of service attack after someone "got unhappy with us."

Motives have changed since then, he argued. "I think it's less malicious, more either business or ignorance like they don't know how to operate." Most damage today is "more like an accident that whoever created a crawler didn't think through how to crawl the internet or they just don't know." Large organisations feel it too: in early October, the Wikimedia Foundation said agents it believes OpenAI operates may have played a part in a partial disruption in May, with bots making up 65% of its most resource-consuming traffic.

"A big farm"

Why has impersonation become urgent now? "I think it's AI," Illyes said. "More and more companies are looking for tokens for training their own models for example and that needs content and the idea is that basically the internet is a big farm where you can just go out and harvest whatever you need."

Firewalls and CDNs, he said, "got better and better identifying these crawlers but they are not perfect. So they still let through some traffic that they meant to stop." Knowing who is crawling "basically enables you to make the decision for yourself about who should access your site and who should not."

The scale behind that remark is visible in Cloudflare Radar data for the seven days ending June 5, 2026, which put bots at 57.4% of HTML traffic, ahead of humans at 42.6%.

Why pretend to be Googlebot specifically? Illyes could only speak to history. "Historically the reason was that the perception was that site owners are more likely to let through Googlebot." He doubts that still holds, "because firewalls in general got much better at identifying spoofers but that doesn't mean that they will catch everything." On the wider internet "you will still see spoofers a lot of spoofers"; behind a firewall, "it gets better."

Who should pay for authentication? "Whoever wants to essentially." Today, he said, that is either a web application firewall provider, "like Cloudflare or Akamai or whatever who historically acted just as a CDN but nowadays they step more and more into the web application firewall business," or site owners who "deploy their own authentication methods."

Robots.txt: "it is a suggestion"

Is robots.txt enough? "I think so," Illyes said, then qualified the answer at length. The file worked "for 20 years or 25 years" as a de facto standard, "meaning that it was not formalized. So basically anyone could interpret it as they wanted to." He placed its formalisation as RFC 9309 "in 2021 or something 23 something like that." The date is 2022: the IETF published RFC 9309 in September of that year, and Google's Search Relations team has written that robots.txt became a standard in 2022.

The standard "gives you very granular control over what can be accessed and what cannot be accessed but as with any non-network level rule set it is a suggestion." A rule telling a crawler to stay out of "the directory that has cute pictures in the name" generally works, he said, "for crawlers that are really big like Bing, Googlebot or Google crawlers in general and so on." Smaller operators are another matter. Some "are not aware of this," and "if you point it out to them generally they are like oh my bad sorry."

That was the reasoning behind Google's open-source work. After the IETF publication, he said, Google released its robots.txt parser and matcher on GitHub "to allow crawler operators to consistently obey robots.txt. So basically they can just plug in our C++ parser into their infrastructure and then they will have the same parsing and matching behavior as any of the large crawler operators."

Crawlers with a reason to ignore it

Can bots just ignore the file? "Sure." Illyes named research crawlers, privacy crawlers and malware crawlers as having "a very valid reason to ignore robots.txt because they are looking for something fishy." The remainder "either they don't know that they should follow robots.txt or they actively don't want to." For those, "that's where web application firewalls jump in and save the day." His summary was blunt: "if you want to block then you need something network level."

Compliance data points the same way. TollBit's report on the first half of 2026 found that about 15% of identified AI page fetchers in Europe reached URLs marked as disallowed.

Fetchers acting for a person

The most contested part of the interview concerned AI agents. Is there a difference between crawlers collecting the web at scale and a bot fetching a page because a user asked? "Absolutely," Illyes said. Crawlers "typically operate independently like there's no human sitting behind," and "because it's automated it needs more safeguards," including obeying robots.txt, "because robots.txt is really for automated recursive clients not individual one fetch."

A fetcher retrieving one page for a person is, in his view, "fundamentally different because technically the user could have gone to that same page and fetch that page with their browser." He flagged the conclusion himself: "this is arguable and I'm happy to argue with the people of the internet but the user could have just gone there and fetched the page themselves. Arguably, there's no good reason to obey robots.txt again because the user could have just gone there and opened that page themselves."

That position matches Google's practice. When Google added Google-Agent to its list of user-triggered fetchers on March 20, 2026, it said the agent ignores robots.txt because a person requested the fetch. OpenAI's documentation similarly states that robots.txt may not apply to ChatGPT-User, because a user initiated the request. Publishers dispute the premise. More than 300 publishing executives lobbied Congress for the Stealth Bot Prohibition Act, introduced in July 2026, which would require AI scraping bots to identify themselves and disclose their purpose.

Illyes's argument treats each fetch as one person's visit. It does not address what an AI service does with the page once fetched, or the combined volume when one assistant serves millions of users. The channel's own pinned comment raised the question at the 18:12 mark: "should an AI agent acting on a user's request follow the same rules as a crawler?"

Asked whether bots should be forced to tell publishers what they do with collected data, Illyes pointed to IETF mailing lists and Internet-Drafts "that are thinking exactly about this." He was clear about the state of play: "I know that there's no consensus on whether this should happen or should not happen. But it is definitely something that we are thinking about."

Search access and AI use, split by one token

Is the web heading towards different access levels for different bots? "Arguably we are already there at least from Google's perspective," Illyes said. Through robots.txt, a site can allow Googlebot to index everything while blocking Google-Extended, "which is a pseudo crawler it's basically just a name." Such a rule tells Google "don't use my content for training AI, not for training, for improving various models, Vertex AI and various Gemini apps or something like that."

He corrected himself mid-sentence, and the wording matters. Google's crawling overview published in March describes the token as controlling whether a site's content helps train future Gemini models, and states that restricting it does not affect inclusion in Search. In 2024, Google's crawler documentation called it a standalone product token for managing whether a site helps improve Gemini Apps and Vertex AI generative APIs.

"So there's already a distinction between AI crawlers and search crawlers that you can use in robots.txt. Apple does the same thing," Illyes continued. Apple's equivalent token is Applebot-Extended; in June 2026, Apple documented it as a separate training control that leaves search indexing unaffected. Illyes expects finer controls "with more and more automated clients on the internet," though "the first step was already made with these dash extended tokens in robots.txt."

There is a tension in his framing. A week earlier, the Google Search Central video described robots.txt as an access protocol that says nothing about what happens to data afterwards. Google-Extended is a use preference carried in that access file. Illyes acknowledged as much by calling it a pseudo crawler that is "basically just a name," and no request ever arrives under it. Infrastructure vendors have built around the split: in September, Cloudflare added a Disallow AI Training option that publishes a no-training preference in robots.txt while keeping Google, Apple and Microsoft's mixed-use crawlers allowed for search. Cloudflare said fewer than 1% of its sites block search bots, against 17% that block AI training in some way.

Web Bot Auth: signatures instead of addresses

The host put it directly: Google is testing signed requests from some AI agents. "So that was an effort that I was kind of involved in early on. Then some other folks took over to actually implement it," Illyes said.

His explanation started with the HTTP request. Metadata such as the user agent travels in the request header, "but that header is not necessarily true." Either "a third party could have intercepted that request and then made some modifications to it for example," or the sender "is a spoofer. So you are pretending to be Googlebot."

Web Bot Auth answers that with "a standard called HTTP signatures that we used to sign cryptographically some parts of the header." Google then commits "to this part being accurate according to our standards and here's our key that you can use to verify our claim." If the check fails, "then you know that okay this is not the real Googlebot or real agent or whatever."

One technical correction applies. Illyes described the site's side as using the key "to decrypt the request." Under HTTP Message Signatures, published by the IETF as RFC 9421, nothing is encrypted: the requester signs selected request components with a private key, and the site uses the published public key to verify that signature. Google's documentation cites RFC 9421 and the IETF Internet-Draft draft-ietf-webbotauth-httpsig-protocol.

What Google actually signs

The documentation also shows how narrow the test is. Google's page "Authenticate requests with Web Bot Auth (experimental)," last updated on October 13, 2026, names a single agent. Only a subset of Google-Agent requests is signed, and those requests are authenticated as https://agent.bot.goog. Signed requests carry a Signature-Agent header with the value g="https://agent.bot.goog", and the public keys are published at https://agent.bot.goog/.well-known/http-message-signatures-directory.

The page states that not all Google user agents use Web Bot Auth and describes Google's implementation as experimental. It recommends continuing to rely on IP addresses, reverse DNS and user-agent strings alongside signatures while signed traffic is rolled out gradually. Googlebot itself - the crawler in the episode's title, and the one Illyes's honeypots catch being impersonated - does not appear on the page as a signer. For now, a fake Googlebot can only be exposed by its address.

Google first disclosed the experiment when it listed Google-Agent in March. Adoption across the industry remains thin. Arkose Labs reported that only 2.3% of the AI agent sessions it classified as legitimate in the third quarter carried a valid Web Bot Auth signature, with ChatGPT agent signing 78.4% of its sessions. The supporting infrastructure has been growing for a year: Cloudflare published a registry format for finding signing keys in October 2025, and in August 2026 said its bot review now checks Web Bot Auth signatures automatically.

Why IP verification has limits

What would signatures let site owners do that they cannot already? "It's not that they cannot do," Illyes said. The problem is that IP addresses are "a limited resource." "We've ran out of IPv4 10 years ago, I think, something like that. So for a new company who comes on the internet and wants to do crawling, it's actually super hard and super expensive to get IPv4 addresses."

IPv6 addresses are much easier to obtain, "but not all the internet accepts still IPv6. So basically some parts of the internet will be dark for you if you only have IPv6." The workaround is sharing an address between a website and a crawler, which leaves publishers guessing: "is this IP address for a site or is this for a crawler?" Reverse DNS is no easier, because "getting access to the reverse zones of the DNS is actually super hard" and zone managers "don't like people tinkering with the reverse zones."

Addresses can also change hands. Google "rotates some of the IP addresses like every now and then new IP addresses come in or fall out from our pool. And then if someone is quick enough to pick up that IP address that fell out of the pool then they could actually in theory at least spoof even the IP address." A signing key carries different stakes: "if we lose our private key for signing the request we have a so much bigger problem," he said. "It would be catastrophic." Keys are therefore "super guarded," making the system "virtually unspoofable."

Bot reputation, still an idea

Could bots have reputations one day? "I would hope so. This is something that I'm toying with. I don't have a internal draft yet or I don't know if I will have an internal draft," Illyes said. He compared it with how people are judged online, where someone with 100,000 followers on LinkedIn or Instagram draws more attention.

Who would decide? "I would imagine that it would be the internet decides." A single arbiter worried him. "If it's like one company deciding that that other crawler is bad or good, then that could be biased." His example was a spam site owner paying a rating company: "here's this $100,000. Tell the world that Googlebot is a bad bot. Then suddenly you have a bias problem." His alternative: "if you let people declare and then you aggregate that declaration somehow and then you use that aggregated declaration as a signal for bot reputation might be interesting." He stressed that "it's still like an idea phase."

A version already exists inside one company. In October, Cloudflare described trust bands for its Verified bots, scored on whether they honour robots.txt and slow down after HTTP 429 responses, with public scores planned but undated. That scheme is run by a single company - the arrangement Illyes said could be biased - although Cloudflare bases its bands on observed behaviour rather than on declarations.

A less open web?

The host asked whether authentication favours established companies. "All of these systems like any one of them have a bootstrap problem," Illyes said. Dedicated IPv4 ranges cost "lots of money"; reverse zones are hard to set up "unless you are good friends with the DNS admin"; and "getting a cryptographic key that is trusted by the root chains" is "also not necessarily easy or cheap." A reputation system would have "similar problems except it would not be a monetary problem. It would be convincing sites that you are a good bot." Newcomers could start "with a base reputation."

He did not expect anonymous crawling to disappear. "I think they will be able to crawl the web. It will be more difficult in some senses but bot owners will always find a way." He acknowledged the criticism: "I can see why some opponents to Web Bot Auth for example would say that this will create a less open internet." Online since "92 93," he concluded: "we always find a way to create crawlers and to crawl the web. That's not going to go away."

For site owners without a security team, his view was that most can wait. "First try to see if you actually have a problem because most sites don't have a problem." Once a site grows, he pointed to a free CDN as the route, and said platforms such as Shopify "have these protections implemented already." For self-hosted WordPress or custom sites, "eventually you probably need to think about it," but early on "bots are not the problem that you need to solve. It's basically market your site first, make it popular and then probably bots will become a problem but later down the line."

Why this matters for marketers

Server logs drive a large share of technical SEO work, from crawl budget diagnosis to checks on whether Google has seen a new page. Illyes's account means that a log line labelled Googlebot proves nothing on its own; only the address checks that Google documents can separate Google Search activity from an impostor, and Google's signatures do not yet cover Googlebot.

The same limit shapes decisions about AI access. Blocking a crawler by its user agent stops only the crawlers that are honest about their names, while allowing Googlebot by name also admits anyone who types it. That matters as publishers weigh offers and terms from AI companies, including Meta's revised terms granting it the right to crawl advertisers' websites for AI, which Meta has delayed to November 20.

Illyes also drew a clear line between crawlers, which in his view must obey robots.txt, and user-triggered fetchers, which have "no good reason" to. Google's own agent is built on the second premise, which limits what robots.txt can do against AI assistants that fetch pages on request. And the token that separates Search from Gemini training, Google-Extended, is a preference about use that works only for operators who choose to honour it.

Caption errors

YouTube's automatic transcript contains errors that affect names and technical terms:

  • Illyes's name appears as "Gary Eish," and the host's surname as "Samkov"; her YouTube handle reads anetasamkoff.
  • The show's name, Odys Podcast, appears as "all this podcast."
  • "Crawling" is rendered as "scrolling" twice, in the opening answer and in "just knowing who's scrolling you," and "crawlers" appears as "crawers" and "corers."
  • The IETF appears as "ITF"; Web Bot Auth appears as "webbot off," "web of thought," "web o" and "web both uh o"; robots.txt appears as "robot txt," "robots cxdt" and "Robots DXT."
  • Googlebot is split into "Google bot" and once into "Google bought"; Mediapartners-Google appears as "media partners Google."
  • Akamai appears as "Akami," Vertex AI as "Vert.Ex AI," IPv4 as "IPV for" and "unspoofable" as "unspoable."
  • Illyes says "Apple extended"; Apple's actual token is Applebot-Extended.
  • The teaser at the start has the host asking about the worst "a known bot" can do; in the full exchange at 10:45 the captions read "unknown bot."
  • A website platform Illyes names alongside Shopify, with a reference to Japan, is captioned as "Amla" and cannot be identified from the captions.

Timeline

Summary

Who: Gary Illyes of Google Search Relations, a former member of Google's crawler team who worked on the robots.txt standard and the early stages of Web Bot Auth, interviewed by host Aneta Samkoff for Odys Podcast: The High Stakes Growth Show.

What: Illyes said changing a crawler's user agent to Googlebot takes "literally one line of code" and about 30 seconds, that impersonation is "very" common based on honeypot sites he runs, and that only published IP ranges and reverse DNS reliably expose fakes today. He called robots.txt "a suggestion," argued that fetchers acting for a single user have "no good reason" to obey it, described Google-Extended as a "pseudo crawler," explained Web Bot Auth's signed requests and floated an early idea for crowd-sourced bot reputation. Google's documentation shows its signing test covers only a subset of Google-Agent requests, not Googlebot.

When: The episode was published on October 28, 2026. Google's Web Bot Auth documentation was last updated on October 13, 2026, and its crawler verification page on March 20, 2026.

Where: On YouTube, in an episode recorded at the Shenzhen SEO conference in China, with implications for any website that receives automated traffic.

Why: AI companies' demand for training data has multiplied the number of crawlers, and many of them identify themselves falsely or not at all. Site owners deciding which bots to admit, and marketers reading Googlebot activity in their logs, depend on knowing which requests genuinely come from Google - a question that user agent strings cannot answer and that cryptographic signatures, so far, answer only for part of one Google agent's traffic.