Googlebot is the automated software Google uses to fetch pages from the public web so they can be processed, indexed and ranked in Google Search. It exists because a search engine cannot return a result for a page it has never retrieved. Every organic listing, every Discover card and every source cited in an AI Overview traces back to a successful HTTP request made by Googlebot to a web server. Publishers and advertisers who never open a robots.txt file still live with what the crawler did, or failed to do, overnight.

A name, not a program

The most consequential revision to how Googlebot is understood arrived in March 2026, when Google engineers Martin Splitt and Gary Illyes explained that Googlebot is not a standalone application. It is one client of a shared internal fetching platform that Google runs, in effect, as a software-as-a-service product for its own teams. Search, News, Shopping, AdSense, Gemini and NotebookLM all call the same infrastructure and receive bytes back. The crawler name attached to a request identifies the requesting product, not a separate piece of code.

"Back then we probably had one crawler because we had one product," Illyes said of the early years, according to the Search Off the Record episode published on 12 March 2026.

The architecture explains behaviour that once looked arbitrary. Rate limiting, retry logic and robots.txt parsing happen once, at platform level, rather than being rebuilt product by product. It also explains why file size caps are expressed as per-product overrides on a platform default rather than as properties of individual bots.

How a crawl proceeds

Google published a plain-language overview of its crawl logic on 3 March 2026. Discovery comes from links on known pages, sitemap files and URL submissions through Search Console. Discovered addresses enter a scheduling queue rather than a simple list, and recrawl frequency follows observed change: a breaking-news homepage may be revisited every few minutes, while a static page can wait a month. Sitemaps inform that schedule without guaranteeing it.

The fetch itself runs over HTTP/1.1 by default, with HTTP/2 selected when crawling statistics favour it; a server can refuse HTTP/2 by answering with a 421 status. Google's crawlers honour conditional requests, sending If-None-Match and If-Modified-Since headers and reading ETag and Last-Modified in response, details added to the crawling documentation in November 2025. Servers under strain can push the crawler back with 429, 500 or 503 responses, though Google's guidance warns that sustained errors beyond one to two days risk URLs dropping out of the index.

HTML that survives the fetch goes to the Web Rendering Service, a headless browser that executes JavaScript so the crawler sees the document object model a visitor would see. Google described that rendering step as costly but applied to all HTML pages. Scripts and stylesheets pulled during rendering are cached for 30 days independently of the HTTP headers those files carry.

The 2MB ceiling

On 6 February 2026 Google cut Googlebot's fetch limit from 15MB to 2MB per URL, a reduction of 86.7%. PDFs keep a separate 64MB allowance, and 15MB remains the platform default for crawlers with no override. The cap applies to uncompressed data and counts HTTP request headers.

What happens at the threshold matters more than the number. Google's documentation, expanded on 31 March 2026, states that the download stops exactly at the cutoff and the partial file is treated as the whole document. Anything past the line is not fetched, not rendered and not indexed. Each resource the renderer requests carries its own separate 2MB counter rather than drawing on the parent page's allowance.

Practical exposure appears narrow. Dave Smart, who added a truncation simulator to the Tame the Bots testing platformthe same week, put the share of affected sites below 0.01%, noting that two megabytes of raw HTML is an extremely large file. The median mobile page weighs about 2.3 megabytes across more than 60 files, up from 816 kilobytes, but HTML is only a fraction of that.

Identity, tokens and verification

Googlebot Smartphone identifies itself with a string ending (compatible; Googlebot/2.1; +http://www.google.com/bot.html), prefixed by an Android 6.0.1 Nexus 5X descriptor and a Chrome version placeholder that tracks the current stable release. The desktop variant carries the same Googlebot/2.1 marker without the mobile fragment. Both answer to the single robots.txt product token Googlebot, so no rule can separate them.

Because user agent headers are trivially forged, Google publishes CIDR ranges in JSON files covering common crawlers, special-case crawlers and user-triggered fetchers. Those files moved to daily refreshes in March 2025 after weekly updates left gaps. On 11 February 2026 they were relocated to a /crawling/ipranges/ path, with a six-month redirect window from the old search-specific location. The manual alternative remains a reverse DNS lookup resolving to googlebot.com, google.com or googleusercontent.com, confirmed by a forward lookup back to the original address. Google states that its crawlers egress primarily from United States addresses. An experimental authentication method, Web Bot Auth, entered the documentation on 4 May 2026.

Crawl budget

Google defines crawl budget as the set of URLs it can and wants to crawl, the product of two variables. The crawl capacity limit caps how long a server spends holding connections open for Google, starts at a conservative default and adjusts to observed response times. Crawl demand reflects site size, update frequency, page quality and relevance. Capacity is shared across all Google crawlers, so heavy AdsBot activity reduces what remains for Googlebot.

Google's own thresholds put the issue beyond most sites: roughly one million unique pages changing weekly, or 10,000 unique pages changing daily. The Search Console crawl rate limiter, the one manual control that existed, was retired on 8 January 2024 on the grounds that it acted slowly and was rarely used.

Origins and evolution

The lineage runs to BackRub, the crawler Larry Page and Sergey Brin operated at Stanford from 1996. Their 1998 paper describing the system detailed distributed crawlers holding around 300 connections open simultaneously. Illyes has dated the surviving infrastructure to a Wget script from roughly 1998 or 1999. The robots.txt convention it obeys predates Google entirely, proposed by Martijn Koster in 1994 and formalised as IETF RFC 9309 only in September 2022.

Mobile crawling arrived in stages, culminating in mobile-first indexing announced in 2016 and completed on 5 July 2024, after which pages unreachable on a mobile device fell out of the index. Rendering became evergreen in May 2019, when Googlebot began tracking current Chromium releases instead of Chrome 41. HTTP/2 crawling began in November 2020. GoogleOther appeared in April 2023 to take non-search fetches off Googlebot's quota, and Google-Extended followed on 28 September 2023 as a robots.txt token for AI training and grounding. Google split the crawler documentation into separate pages in September 2024, and added Google-Agent on 20 March 2026 for AI systems browsing on a user's behalf.

Why it matters, and where it is contested

Googlebot's dual role is the centre of the current dispute. The same fetch that produces a blue link also supplies content to AI Overviews and AI Mode, and the Google-Extended token does not stop it, since blocking Googlebot removes a site from Search altogether.

Cloudflare has pressed that point with volume data. Chief executive Matthew Prince argued in January 2026 that Google sees far more of the web than any AI rival, citing 3.2 times the coverage of OpenAI and 4.8 times that of Microsoft; over a two-month sample Googlebot reached 1.7 times more unique URLs than ClaudeBot, 1.76 times more than GPTBot, 166.98 times more than PerplexityBot and 714.48 times more than CCBot. Voluntary standards have found little traction: Google's John Mueller said in July 2026 that no crawler he knew of honoured Cloudflare's content-signal directives, a position that sharpened publisher scepticism about robots.txt as a remedy. Illyes had earlier outlined the consensus processrequired to extend the protocol, calling unilateral change impossible by design.

Regulation moved first. On 3 June 2026 the UK Competition and Markets Authority imposed a binding conduct requirement giving publishers opt-outs from AI Overviews, AI Mode, Gemini and the Vertex AI API without losing search visibility, with main obligations due on 3 December 2026 and page-level grounding controls on 3 March 2027.

Disambiguation

Googlebot and GoogleOther are separate tokens on the same platform. GoogleOther handles internal research and non-search fetches so that work does not consume Googlebot's crawl allowance.

Google-Extended is a robots.txt control, not a crawler with its own user agent string. It governs whether fetched content may be used for Gemini training and grounding; it does not stop the fetch.

AdsBot is a special-case crawler that checks landing page quality for advertising products and may operate under a site agreement rather than robots.txt alone.

Google-Agent is a user-triggered fetcher, not a crawler. It acts only when a person invokes an AI agent, and by design it ignores robots.txt. GPTBot, OpenAI's training crawler, sits in a different category again: it never indexes for a search results page.

Recent developments

Google's crawling changelog shows steady consolidation through 2026. On 16 July the NotebookLM user agent became Google-GeminiNotebook after the product was renamed, and on 22 July the crawl budget guide was rewritten for consistent terminology. On 28 July Illyes confirmed that the unavailable_after directive still requires a full re-crawl to take effect, since Google rarely issues HEAD requests, limiting its value where expired listings generate heavy 404 traffic. On 31 July Google dropped the 2007-era requirement that internal search results pages be blocked from crawlers, reclassifying the practice as inefficiency rather than a policy breach.

Timeline

  • 1994: Martijn Koster proposes the robots exclusion convention
  • 1996: BackRub crawler begins operating at Stanford University
  • 1998: Brin and Page publish the paper describing Google's distributed crawler architecture
  • April 2023: GoogleOther introduced to offload non-search fetches
  • 28 September 2023: Google-Extended token launched for AI training and grounding controls
  • 8 January 2024: Search Console crawl rate limiter tool removed
  • 5 July 2024: Mobile-first indexing completed
  • September 2024: Crawler documentation split into per-crawler pages
  • March 2025: Crawler IP range files switch from weekly to daily updates
  • November 2025: Crawling documentation expanded with caching, protocol and throttling detail
  • 3 February 2026: File size limits moved to the crawling documentation site
  • 6 February 2026: Googlebot fetch limit cut from 15MB to 2MB per URL
  • 11 February 2026: IP range files relocated to the /crawling/ipranges/ path
  • 3 March 2026: Crawling overview page published
  • 12 March 2026: Google engineers describe the shared crawling platform on Search Off the Record
  • 20 March 2026: Google-Agent added as a user-triggered fetcher
  • 31 March 2026: Byte limits and crawler categories documented in two blog posts
  • 4 May 2026: Web Bot Auth documented as an experimental verification method
  • 3 June 2026: UK CMA imposes AI opt-out conduct requirement on Google
  • 16 July 2026: NotebookLM user agent renamed Google-GeminiNotebook
  • 22 July 2026: Crawl budget guide revised
  • 31 July 2026: Requirement to block internal search pages removed from Search Essentials

Summary

Who: Google operates Googlebot through a shared internal crawling platform used by Search, News, Shopping, AdSense, Gemini and other products, with Gary Illyes and Martin Splitt among the engineers who publicly document it. Site owners, publishers and technical marketers manage its access through robots.txt, server responses and Search Console.

What: An automated fetching client that requests pages over HTTP, passes HTML to a headless rendering service and supplies the resulting content to Google's indexing systems. It reads the first 2MB of a supported file and the first 64MB of a PDF, identifies itself with a Googlebot/2.1 user agent string, and can be verified through reverse DNS or published IP ranges.

When: Descended from the 1996 BackRub crawler and infrastructure dating to about 1998, with mobile-first indexing completed in July 2024, the fetch limit cut to 2MB in February 2026, and the shared-platform architecture disclosed in March 2026.

Where: Requests egress primarily from United States addresses, with documentation, IP range files and the crawling changelog published on Google's developer site.

Why: Search results cannot exist without retrieval, so Googlebot determines what enters the index and, increasingly, what feeds AI Overviews and AI Mode. That dual purpose is now the subject of regulatory intervention in the United Kingdom and competitive complaints from infrastructure providers, because publishers cannot refuse the AI use without also refusing search.