Google revamps documentation for crawlers and user-triggered fetchers
Google revamps crawler documentation, adding product impact info and robots.txt snippets for each crawler user agent.
Google today made a significant update to its documentation for Google crawlers and user-triggered fetchers. The search giant has reorganized and expanded its documentation to provide more detailed information about how each crawler affects various Google products and services.
The update, which was published on the Google Search Central documentation site, comes as part of Google's ongoing efforts to improve transparency and assist webmasters in managing their sites' crawl preferences. According to Google, the documentation had grown lengthy, limiting their ability to extend the content about crawlers and user-triggered fetchers.
Key Changes in the Documentation
- Reorganization: The previously consolidated information has been split into several pages for easier navigation and comprehension.
- Product Impact: Each crawler now has a dedicated section explaining which Google products are affected by crawling preferences addressed to that specific user agent.
- Robots.txt Examples: Google has added sample robots.txt snippets for each crawler, demonstrating how to use the user agent tokens effectively.
- Content Encoding Support: The documentation now includes information about the content encodings (compressions) supported by Google's crawlers and user-triggered fetchers.
- User Agent String Update: The URL in the GoogleProducer HTTP user agent string has been updated to match the value used by the actual fetcher.
Notable Crawler Effects
- Googlebot: Affects Google Search, Discover, Google Images, Google Video, and Google News.
- Googlebot Image: Impacts Google Images, Discover, Google Video, and image-related features in Google Search.
- Googlebot News: Influences all surfaces of Google News, including the News tab in Google Search and the Google News app.
- Google StoreBot: Affects all surfaces of Google Shopping, including the Shopping tab in Google Search.
- Google-InspectionTool: Impacts Search testing tools like the Rich Result Test and URL inspection in Search Console.
- Google-Extended: A new standalone product token for managing site participation in improving Gemini Apps and Vertex AI generative APIs.
The documentation also covers special-case crawlers like AdsBot and AdSense, which have specific roles in ad quality checking and serving relevant ads.
Google emphasized that these changes are primarily documentation updates and do not reflect any alterations in crawler behavior. The company stated, "There were no meaningful changes to the content otherwise."
This update is particularly relevant for SEO professionals, webmasters, and developers who need to manage how Google crawlers interact with their websites. By providing more detailed information about each crawler's impact, Google aims to help site owners make informed decisions about their crawl preferences and optimize their sites for various Google products.
The revised documentation is now live on the Google Search Central website. Webmasters and SEO professionals are encouraged to review the updated information to ensure their sites are configured optimally for Google's various crawlers and user-triggered fetchers.
As search technologies continue to evolve, staying informed about these updates is crucial for maintaining visibility and performance across Google's ecosystem of products and services.
Key Facts
- Announcement Date: September 16, 2024
- Documentation Location: Google Search Central
- Main Changes: Reorganization, product impact details, robots.txt examples
- New Feature: Google-Extended user agent for AI and generative API improvements
- No changes to actual crawler behavior