On November 11, 2026, Google published guidance on a problem that, by its own account, catches site owners at the very moment they think they have solved their capacity troubles: crawler traffic to the origin server spikes right after a site moves behind a content delivery network. The post on the Google Search Central Blog, titled "Moving to a CDN without overwhelming your origin (or blocking Googlebot)," arrived alongside the fifth and final episode of the first season of The Non-Canonical Cabinet, a YouTube series hosted by Gary Illyes and Cherry Prommawin. Both pieces reach the same uncomfortable conclusion. The instinctive response to the spike, switching on blanket bot protection in the CDN dashboard, can lock out Google's crawlers and, according to the video, make products disappear from Google Shopping and stop ads products from working.

In Short

Google says moving a website onto a content delivery network can briefly flood the original server with crawler requests, because a faster site invites faster crawling while the network's caches are still empty. Site owners who react by switching on blanket bot blocking can cut off Google's crawlers, which Google says pushes pages out of Search and, according to its video, products out of Shopping and ads. Google's own fixes are to let its verified crawlers through, fill the caches before launch and, if the server is struggling, send 429 or 503 responses for no more than a day or two.

What Google published

A CDN copies a site's assets - HTML documents, CSS, JavaScript and images - to edge servers spread around the world, closer to visitors. According to Google, that lowers visitor latency and "shields your origin server (the backend source of truth holding your original content) from heavy traffic." For a growing site, the blog calls it "a major upgrade."

Then comes the twist. "We sometimes hear from site owners that right after putting their site behind a CDN, crawler traffic to their server spikes almost immediately, which is exactly what they were trying to fix," the post states. "Before those edge servers populate their caches, both user visits and crawler requests still need to be fetched directly from the origin."

The post is dated Wednesday, November 11, 2026, and carries author links for Prommawin and Illyes. Its sign-off credits "Cherry, Gary, empty-cache chicken, a bubble-wrapped chicken, and an alarm blaring in the background," which gives a sense of the series' tone: network architecture explained with a plastic construction hardhat, a smoking filing cabinet and a rubber chicken standing in for an edge server. The accompanying video, "Moved to a CDN: Why Is Everything Crawling Everywhere All at Once?", runs 9 minutes and 17 seconds on the Google Search Central YouTube channel. Its description identifies Illyes and Prommawin as Google Search Developer Advocates and links to the blog post, to Google's page on reducing Googlebot's crawl rate and to its crawl budget guidance for large sites.

Google gives no figures for how often the problem occurs, how large the spikes get or how long they last. The post describes a mechanism, not a dataset.

Why a faster site gets crawled harder

The explanation starts with what Google calls the crawl capacity limit, also known as hostload. According to the blog, Google's automated crawling infrastructure continuously monitors the health and responsiveness of a host server. When response times increase, crawlers slow down so they do not overwhelm the server or degrade the experience for human visitors. When responses are fast and consistent, the systems read that as healthy, capable infrastructure. "Provided there is crawl demand, the crawl capacity limit gradually goes up."

A CDN changes that signal almost overnight. "The moment your DNS points to a fast CDN edge network, connection times and response times drop dramatically," the blog says. "To our crawling systems, it looks like your server just got a massive hardware upgrade, so our crawlers start ramping up their request rate to match."

The video puts it more bluntly. "Basically, CDNs take the valve off the hose. So now crawlers are drinking from a fire hose," one of the hosts says. Earlier in the episode, the rule the crawlers normally follow is spelled out: "When a server responds faster, the crawlers assume the infrastructure is healthy and slowly ramp up speed." The other half of that rule is described too: "Our automated systems constantly monitor server health, and when a site slows down, our crawlers back off so we don't crash the site."

Google's crawl budget documentation, last updated on July 22, 2026, fills in details the blog leaves out. According to that page, every site starts with the same conservative default capacity limit; the limit rises when latency and time to first byte are stable or improving and falls when a site slows down or returns 5xx or 429 responses; and it is shared across all of Google's crawlers. The page is written for large sites - roughly 1 million or more unique pages changing about weekly, or 10,000 or more changing daily - though Google describes those thresholds as rough estimates.

The cold cache problem

Faster crawling would not hurt anyone if the CDN absorbed the extra requests. At first, it does not. "The catch is that a brand-new CDN starts with an empty (cold) cache," according to the blog.

On the first request for any URL at a given edge location, the CDN holds no copy. That is a cache miss. The edge server has to "walk" back to the origin, fetch the response, store a cacheable copy and only then serve the client. Only later requests for the same URL become cache hits served from the edge without touching the origin. "Basically, the real bottleneck during a CDN launch is still the origin server," the post states.

The difficulty lies in who is asking. "Human visitors usually cluster around a relatively small set of popular pages (your homepage, top articles, or active promotions) which warm up in the CDN cache quickly," Google writes. "Crawlers do the exact opposite. Their job is to systematically explore the breadth of your URL inventory."

Combine the two behaviors and the numbers turn against the origin. Fast edge responses tell Google's crawlers that faster crawling is safe, while each uncached URL they reach passes straight through to the origin server. The video offers a hypothetical. If a crawler starts working through 100,000 unique URLs as it gradually increases its speed, "every single one of those unique first time requests" misses the cache, "which means your origin server may get slammed by the flood anyway."

Google does qualify the warning. Crawlers do not ramp up their rate immediately - "It won't happen in a millisecond," one host says in the episode - and, according to the blog, many CDNs include built-in connection pacing that smooths out spikes. The video describes this as rate limiting that "can already help pace the flow while the edge servers get up to speed." Even so, "until your edge caches are populated (warmed up), your CDN isn't acting as a shield yet, your origin server is still doing the heavy lifting," the post says.

The pattern is not unique to Google's crawlers. In April, Cloudflare and ETH Zurich argued that AI bots are breaking the web's cache layer, reporting that automated sources make up 32% of traffic on Cloudflare's network and that AI agents iterating on a query show unique-access ratios of 70% to 100%. That is the same long-tail access pattern that leaves caches built for human browsing with little to serve.

When bot protection locks out Google

The second half of the guidance concerns the reaction rather than the cause. An unexpected surge of automated requests in origin logs right after a migration, the blog concedes, "can understandably make admins nervous." The video describes what happens next: "They look at the spike in traffic, assume they are being aggressively scraped by botnets, and naturally they panic." The next step is a login to the CDN dashboard and a switch to blanket bot protection. The blog sums up the mood with a line it attributes to Prommawin: "BOTS SHALL NOT PASS"!

"The problem with blanket security rules is right there in the name," according to Google. Without exemptions, those rules treat all automated traffic the same way, including traffic from crawlers that site owners want. When a CDN or web application firewall flags automated requests and intercepts them with a challenge such as a CAPTCHA, Google's crawlers fail the test. "Googlebot has many talents; spotting traffic lights and chickens is not one of them," the post says.

The video adds a technical point the blog only implies. Blocked crawlers, one host explains, hit "a hard 403 Forbidden wall or you've been blocked message with a 200 status code, and our crawlers cannot know it's a security check. It just looks like the entire website is now broken or intentionally hidden." A challenge page served with a 200 code, which normally signals success, tells a crawler nothing about why the expected content is missing.

From there, the post stops being a networking lesson. "Leaving blanket bot protection enabled without exempting verified crawlers leaves crawlers basically stuck on an error page that's not useful for Search, or other products relying on crawled data," the blog states. "From our perspective, your website appears deliberately restricted or broken. If left in place, our crawlers will back off, and your pages, shopping listings, and your chicken pictures will begin to drop out of Search."

The video is more explicit about the commercial effect. "If a site leaves bot protection on without an explicit exemption rule, Google's crawlers will back off entirely because the site looks like it's offline," the hosts say. "We drop its pages from the search index, products disappear from Google Shopping and ads products stop working."

Neither source says how quickly any of this happens, or how long a block has to stay in place before pages or products are removed.

Why Shopping and ads are exposed

For advertisers, the reference to Shopping and ads follows from how Google's crawling is built. In March, Illyes and Martin Splitt described Googlebot as one client of a shared internal crawling platform, naming Google Shopping and AdSense among the products that draw on the same infrastructure. Google's crawl budget page adds that the capacity limit is shared across all crawlers, and that crawl demand runs higher for AdsBot when advertisers use dynamic ad targets and for Google Shopping when products are listed in merchant feeds.

The same CDN rule that challenges Googlebot can therefore challenge the crawlers that check Google Ads landing pages and Merchant Center product pages. In March, PPC Land reported on Merchant Center's "product page unavailable" error, which pauses Shopping ads when Google's crawler times out or fails to resolve a merchant's site, with approval returning about 24 hours after a manual website check. A firewall rule that turns crawler requests into challenge pages reaches a similar end from the other direction: the products are there, but Google cannot see them.

Google's documentation on reducing crawl rate, last updated on October 6, 2026, also notes that a reduced crawl can affect Google Ads campaigns.

What Google lists as the fix

The post frames its recommendations as a game plan in three parts, opening with the line that no one needs "to employ a fire marshal to deploy a CDN."

The first concerns verified bots. "Most major CDNs and WAFs offer a built-in toggle or managed rule to allow known / verified bots," according to the blog, which lets chosen crawlers bypass CAPTCHA challenges while unwanted traffic is still filtered. Google points to its published IP ranges as the way to identify its crawlers. Its verification page, last updated on March 20, 2026, describes reverse and forward DNS lookups against googlebot.com, google.com and googleusercontent.com, plus five JSON files of IP ranges, files that moved to a new location on March 31. The blog goes further and calls it "better yet" to configure robots.txt or rules that target specific unwanted bots "rather than blocking all automated clients." In the video, the allow-known-bots setting is described as "a good solution."

The second is pre-warming. Before routing all traffic through a new CDN, Google describes programmatically fetching the highest-traffic pages, upcoming peak-traffic pages "such as promotional or seasonal launches" and the most frequently crawled URLs through the CDN at a controlled pace, so edge caches are ready. The post adds, in keeping with the series: "Or, if unlike Gary you actually have friends, ask them to access two distinct URLs each; that works too."

The third is a temporary brake. If crawling still puts critical load on the origin while caches warm, Google describes setting server or CDN rate-limiting rules to return a 429 (Too Many Requests) or 503 (Service Unavailable) status code "for a short window (a few hours, or 1-2 days at most)." In the video's words: "When the server is overwhelmed, our crawlers will slow down momentarily."

Where the post and the episode differ

The two sources are not word for word on status codes. The video speaks of returning "a 429 or 500 class HTTP status code"; the blog narrows that to 429 and 503. Google's reduce-crawl-rate page lists 500, 503 and 429, frames the measure as lasting from a couple of hours to one or two days, and warns that URLs which keep returning those codes for several days may be dropped from Google's index. On the failure mode, the blog refers to 403 walls and challenge pages, while the video specifically mentions block pages served with a 200 code. The video also calls rate limiting in the CDN admin interface an alternative to the status codes, where the blog presents rate-limiting rules as the way to send them.

The episode's recap compresses everything into three points: faster response times signal that much faster crawling is safe; CDNs serve from cache only on the second visit, because the first one hits the origin; and the setting that allows known and trusted bots keeps a CDN firewall from accidentally blocking Google's crawlers.

Old mechanics, new pressure

None of the mechanics is new. Google retired the manual crawl rate limiter in Search Console on January 8, 2024, leaving server responses as the fastest way to slow Googlebot down. Its November 2025 documentation overhaul set out that 500, 503 and 429 responses reduce crawling across entire hostnames and should not be served for longer than one to two days. The Search Central Blog ran a post titled "Crawling December: CDNs and crawling" in December 2024. And on October 21, Google Search Central told sites patching PHP to serve 503, not 404, during downtime.

Google's crawling on CDN-backed hosting has drawn scrutiny before, too. In August 2025, crawl rates fell across sites on Vercel, WP Engine and Fastly, a drop Google attributed on August 28, 2025, to an issue on its side.

What has changed is the climate around bot controls. Cloudflare chief executive Matthew Prince said that automated traffic passed human traffic on Cloudflare's network in May 2026. In October, the Wikimedia Foundation said agents it attributes to OpenAI may have contributed to a partial outage, and that bots account for 65% of its most resource-heavy traffic. Operators have more reason than ever to reach for blocking tools, and the tools themselves have become more assertive. Cloudflare's original design for its September 15 AI training controls would have blocked Googlebot on sites that had opted out of AI training, a plan it later dropped. On October 16, Cloudflare said it would score Verified bots on whether they follow robots.txt and slow down after 429 responses, the same status code Google now names as a brake.

The guidance leaves one tension unresolved. Google asks site owners to recognize its crawlers and let them through, at a time when the volume of automated traffic gives those owners every incentive to block first. The remedy Google offers depends on CDN vendors maintaining accurate verified-bot lists and on operators configuring them correctly, two steps outside Google's control.

The end of season one

The CDN episode closes the first season of The Non-Canonical Cabinet. The series opened on October 14 with characters in domain names and URLs, followed on October 21 by an episode on what a robots.txt file can contain. According to the blog, the five episodes also covered 404s and redirects. A second season is, in the post's words, "budget pending," and the hosts ask viewers for edge cases to pull "from the (slightly toasted) cabinet." As one of them puts it in the episode: "When I applied for this role, there was nothing in the job description about explaining networking with chickens."

Why this matters for marketers

For performance marketers, the post is a reminder that infrastructure work owned by engineering or IT teams can reach paid media without warning. CDN migrations are usually justified on speed, and the speed case is real: Cloudflare's Smart Shield launch in September 2025 cited 91.7% of cache-served pages with good time to first byte, against 79.7% for pages processed at the origin. Google's post shows the transition carries its own risks on two fronts. The origin can be overwhelmed by crawling before the caches warm, and the defensive reflex can cut off the crawlers that feed Search, Shopping and ads.

Timing adds weight. Google's own list of pages to pre-warm includes promotional and seasonal launches, and the post lands in the run-up to Black Friday and Cyber Monday, when merchants lean heavily on product listings staying live.

What the guidance does not provide is just as telling. Google publishes no data on how much crawling increases after a CDN move, how long a typical cache takes to warm, or how long a block can last before pages and products drop. Without those thresholds, site owners are left to judge the risk from their own server logs and Search Console reports.

Timeline

Summary

Who: Google Search Central, through a blog post and a YouTube episode presented by Gary Illyes and Cherry Prommawin, described as Google Search Developer Advocates. The guidance affects site owners, developers and merchants who put their sites behind a CDN, and the advertisers whose Shopping listings and ads depend on Google's crawlers reaching those sites.

What: Guidance explaining that moving to a CDN can raise Google's crawl rate because faster responses lift the crawl capacity limit, while cold caches send every first request for a URL to the origin server. Google warns that blanket bot protection without verified-bot exemptions blocks its crawlers, which the blog says leads pages and shopping listings to drop out of Search and the video says makes products disappear from Google Shopping and stops ads products from working. Google lists three measures: allow verified bots, pre-warm caches, and return 429 or 503 for a few hours or 1-2 days at most.

When: November 11, 2026, as the season one finale (episode five) of The Non-Canonical Cabinet.

Where: The Google Search Central Blog at developers.google.com/search/blog/2026/11/crawling-cdn and the Google Search Central YouTube channel, with supporting pages in Google's crawling infrastructure documentation.

Why: Google says site owners regularly report crawler spikes right after a CDN move and sometimes respond with blanket bot blocking. The post explains why the spike happens and why blocking it can cost a site its Search visibility and, according to the video, its Shopping and ads presence. It gives no data on how often this happens or how quickly pages and products are removed.