Google Search Console crawl rate limiter tool to be deprecated

Google last week announced the Crawl Rate Limiter Tool in Google Search Console will be deprecated on January 8, 2024.

The Crawl Rate Limiter Tool in Google Search Console was a feature that allowed site owners to control how often Googlebot crawls their website.

This tool has been available for over a decade, but with advancements in crawling logic and other tools, its usefulness has diminished.

Googlebot, Google's web crawling bot, reacts to how a website's server handles its HTTP requests. If the server consistently returns HTTP 500 status codes or if response times get significantly longer, Googlebot will automatically slow down crawling.

The Crawl Rate Limiter Tool had a much slower effect on crawling speed, and it was rarely used by site owners. With the deprecation of this tool, Google is setting the minimum crawling speed to a lower rate, ensuring that crawlers don't waste a site's bandwidth.

Google is also keeping the Googlebot report form for reporting unusual Googlebot activities and for emergency cases. The fastest way to reduce crawl rate is to instruct Googlebot through server responses.