Google AdsBot is the automated client Google Ads uses to fetch advertiser landing pages. It exists because a paid click sends a person somewhere, and the auction needs to know about that destination first: whether the page loads, whether it matches what the advertisement promised, whether it breaks a policy, and in some campaign types what the page is about at all. Every disapproval labeled "Destination not crawlable" traces back to a request AdsBot could not complete.

A crawler that plays by different rules

Google sorts its automated clients into three categories. Common crawlers, of which Googlebot is best known, always respect robots.txt for automatic crawls. User-triggered fetchers act when a person asks for something and generally ignore it. Between them sit the special-case crawlers, used by products where the crawled site and the product have an arrangement about how the crawl proceeds. AdsBot is the standing example Google gives, and PPC Land documented the split when Google rewrote its crawler documentation in March 2026.

One line of robots.txt behavior surprises people every year. AdsBot ignores the global user agent group, so a file opening with a wildcard user agent and a blanket disallow has blocked Googlebot and left AdsBot untouched. Google treats the advertiser's decision to submit a URL as the permission that matters, and blocking the crawler requires naming it. Google's Display & Video 360 documentation spells out the trap in reverse too: a group naming AdsBot-Google with an empty allow directive blocks the crawler, while the same group with an empty disallow permits it.

Tokens, strings and retired agents

Two agents are currently documented. The desktop crawler identifies itself as AdsBot-Google (+http://www.google.com/adsbot.html) and answers to the token AdsBot-Google. The mobile web crawler sends an Android 6.0.1 Nexus 5X string with a Chrome version placeholder, closing with AdsBot-Google-Mobile. Google's special-case reference, last updated on February 11, 2026, lists two retired ones: an iPhone variant of AdsBot-Google-Mobile, and AdsBot-Google-Mobile-Apps, which checked Android app page quality and obeyed AdsBot-Google rules while ignoring the wildcard.

Page-level control exists separately. A meta tag naming AdsBot-Google with a noindex value stops the crawl for that page, a configuration Google's troubleshooting instructions flag when a dynamic ad target is rejected.

Verification matters because a user agent string is voluntary text any scraper can copy. Special-case crawlers egress from a different address range, published as a JSON object at the /crawling/ipranges/ path since February 11, 2026, and their reverse DNS mask resolves to rate-limited-proxy-***-***-***-***.google.com rather than the googlebot.com pattern.

What triggers a fetch, and how often

Three events put AdsBot on a server: an advertisement enters review and its destination is checked, a dynamic ad target is created that points the crawler at a set of pages rather than one URL, and existing destinations are rechecked.

Google's published figures for that schedule disagree. The Search Console crawl stats documentation says AdsBot crawls URLs roughly every two weeks. The Search Central page on crawling errors, describing the same behavior, says the crawl driven by Dynamic Search Ads targets recurs every three weeks. Neither figure has been reconciled publicly.

The crawler inherits the platform properties: HTTP/1.1 by default with HTTP/2 when crawl statistics favor it, gzip, deflate and Brotli encodings, and a 15MB default file ceiling that individual products may override. Google cut Googlebot's ceiling to 2MB in February 2026 but has published no AdsBot override. PPC Land's March 2026 analysis treated the 2MB cutoff as a risk for script-heavy landing pages; Google's crawler overview, updated on June 12, 2026, still names 15MB as the default and only Googlebot as carrying less.

One behavior is specific to the crawler. Google documents that AdsBot is more likely than other clients to send the conditional request headers If-Modified-Since and If-None-Match, letting a server answer with a 304 status and an empty body when nothing has changed. Across a large catalog, that is the difference between a routine recheck and a bandwidth event.

What the crawl decides

Three systems consume the output.

The first is ad quality. Landing page experience is one of the three Quality Score components, alongside expected clickthrough rate and ad relevance, each rated above average, average or below average against advertisers whose ads appeared for the same exact search over the previous 90 days. Google calls Quality Score a diagnostic on a one to ten scale rather than an auction input, and states that crawler information may feed the assessment. The Google Ads API exposes the signal as post_click_quality_score.

The second is policy. Under destination requirements, a destination that cannot be crawled is a violation in its own right, with disapproval reasons enumerated in Google's advertising APIs as destination not crawlable, destination not working and destination mismatch. The policy page names blanket robots.txt exclusions and insufficient crawl capacity as examples, and promises a warning at least seven days before suspension. The same framework covers the thin intermediate destinations PPC Land examined. Operational limits sit alongside: fewer than ten redirects, click-through URLs between 12 and 2,048 characters, no sign-in wall, no CAPTCHA, no geographic block.

The third is targeting, where AdsBot stopped being a referee and became an input. Dynamic Search Ads use the crawl to index a site and match live queries to pages, generating the headline from page content, with the HTML title named as the most important signal. Advertisers steer the crawler through page feeds, capped at 100 per account, or through URL_Equals and URL_Contains targets and Google-generated category sets.

Origins and evolution

Google has never published a launch date for AdsBot. It appears in third-party crawler inventories by 2008, the year the destination became a measurable quality factor: Google announced on March 6, 2008 that landing page load time would enter Quality Score, and the metric began affecting keyword scores that June. Measuring load time requires fetching the page, and the crawler was the instrument.

The October 2011 Dynamic Search Ads beta changed its role from inspection to targeting, a shift reinforced when Google rebuilt the format and released it globally in 2015. Documentation followed the product: Google split its crawler reference into per-agent pages on September 16, 2024, then migrated the material to a dedicated crawling infrastructure site on November 20, 2025.

Limitations and disputes

The asymmetry in robots.txt handling is the oldest complaint. John Mueller of Google confirmed in 2022 that AdsBot does not follow the standard directives and must be addressed by name, advising site owners drowning in requests to file a crawl rate report rather than edit their exclusion file.

Load is the second. Google's guidance on reducing crawl rate lists Dynamic Search Ads targets among the three most common causes of a sudden crawl increase, alongside faceted navigation and calendar URLs, and offers two remedies: cut the ad targets or buy more server capacity. The same page warns that throttling Google with 429, 500 or 503 responses reaches Google Ads as well as Search, and that campaigns may be paused or cancelled as a result.

Measurement is thin. The Search Console crawl stats report breaks activity out by AdsBot as a crawler type, but only for verified owners of a root-level property, which an agency running the account frequently is not. Nothing inside Google Ads shows an advertiser when its destinations were last fetched.

Network-edge enforcement adds a newer risk. Cloudflare data cited by PPC Land in January 2026 showed sites blocking AI crawlers roughly seven times more often than Googlebot and Bingbot, and from September 15, 2026 the company will block Training and Agent crawlers by default on advertising-carrying pages for domains newly joining its network. AdsBot falls into neither category, classified by Cloudflare as ads verification and by IAB Australia's August 2026 guidance as operational and advertising infrastructure. Misclassification at the edge, rather than deliberate blocking, is the more plausible failure mode.

Consent is a live question too. Google told advertisers in 2026 that terms effective July 1 clarify how submitted inputs are used, echoing Local Services Ads terms updated in March 2025 granting rights to access, index, cache or crawl submitted URLs. The advertiser grants that permission; the site owner, potentially a different party, is not asked.

Disambiguation

Googlebot crawls for Search, always respects robots.txt including the wildcard group, and resolves to googlebot.com addresses. Allowing one says nothing about the other.

Mediapartners-Google is the AdSense crawler. It visits pages carrying AdSense code so Google can match advertisements to content, serving publishers rather than advertisers.

Google-Extended is a robots.txt control token with no crawler of its own. It governs whether fetched content trains or grounds Gemini, and has no bearing on advertising.

OAI-AdsBot is OpenAI's equivalent, documented in April 2026 to validate landing pages submitted as advertisements on ChatGPT and to judge when one is relevant. A deliberate parallel, built for a competing marketplace.

Recent developments

The largest change to AdsBot is not to the crawler but to what consumes it. Google declared AI Max for Search out of beta on April 15, 2026 and announced the retirement of Dynamic Search Ads, whose landing-page-only logic gives way to a broader signal set drawing on ad copy, keywords and real-time intent. Practitioners pushed back through May 2026, arguing that final URL expansion routes specific product queries to generic pages where the older crawl-and-match mechanic did not. Google then moved the migration deadline, and its help documentation as of August 2026 states that DSA campaigns upgrade automatically starting in February 2027, while broad match and automatically created asset campaigns face the earlier September 1, 2026 date. The crawl survives the format. Final URL expansion still requires Google to know what sits on every non-excluded page of a domain.

Timeline

  • March 6, 2008: Google announces that landing page load time will become a Quality Score factor
  • June 2008: Landing page load time begins affecting keyword Quality Scores
  • October 2011: Dynamic Search Ads launch in beta, turning the ad crawl into a targeting mechanism
  • July 2015: Dynamic Search Ads rebuilt and made available to advertisers globally
  • September 16, 2024: Google splits crawler documentation into per-agent pages with robots.txt examples
  • November 20, 2025: Crawling documentation migrates to a dedicated crawling infrastructure site
  • December 18, 2025: Crawl rate reduction and crawl budget guidance move to the new site
  • February 3, 2026: Default file size limits relocated to the crawling documentation
  • February 11, 2026: Special-case crawler reference updated and IP range files moved to /crawling/ipranges/
  • April 15, 2026: AI Max for Search leaves beta; Dynamic Search Ads scheduled for retirement
  • June 12, 2026: Crawler overview page updated, retaining 15MB as the platform default
  • June 2026: DSA automatic upgrade deadline moved to February 2027
  • September 15, 2026: Cloudflare default block on Training and Agent crawlers for ad-carrying pages takes effect for newly onboarded domains
  • February 2027: Scheduled automatic upgrade of Dynamic Search Ads campaigns to AI Max

Summary

Who: Google operates AdsBot as part of the shared crawling infrastructure behind Search, Shopping, AdSense and Gemini. Advertisers trigger it by submitting destinations; site owners, who are often a different party, absorb the traffic and control access through robots.txt and server configuration.

What: A special-case crawler that fetches ad landing pages to check availability and policy compliance, to inform the landing page experience component of Quality Score, and to index site content for dynamic targeting. It answers to the AdsBot-Google and AdsBot-Google-Mobile tokens, ignores the wildcard robots.txt group, and can be verified through published IP ranges and a rate-limited-proxy reverse DNS mask.

When: Documented in third-party crawler references from 2008, the year landing page load time entered Quality Score. Its targeting role began with the Dynamic Search Ads beta in October 2011 and is being reshaped by the AI Max migration running from September 2026 to February 2027.

Where: Requests originate primarily from United States addresses across a distributed crawling fleet, with reference documentation published on Google's crawling infrastructure site and enforcement rules in the Google Ads and Advertising Policies help centers.

Why: An advertisement is a promise about a destination, and the only way to test that promise at scale is to fetch the page. That makes AdsBot the mechanism through which a server configuration decision becomes a media outcome, and the reason a crawler blocked for one purpose can stop campaigns for another.