Ahrefs updates its Site Audit location to United States

Ahrefs has shifted its Site Audit crawler operations from France to the United States, marking a substantial change in its technical architecture. The modification, which took effect on December 23, 2024, introduces new IP ranges and retires several existing ones.

According to the company's technical documentation, the transition affects the AhrefsSiteAudit crawler, which previously operated from France. The geographical relocation aligns the platform's crawling infrastructure more closely with Google's primary crawling location, potentially offering more accurate representations of how search engines interact with websites.

The technical implementation includes the introduction of a new IP range, 202.8.40.0/22, which network administrators and technical teams need to add to their whitelists to maintain uninterrupted site crawling capabilities. This addition comes alongside the retirement of multiple IP addresses, creating a significant shift in the platform's networking infrastructure.

The company has already completed the retirement of the 168.119.* IP addresses, which are no longer active in the system. Furthermore, the 195.154.* IP addresses will cease operations on December 31, 2024. Technical teams managing website configurations have until this date to maintain these IPs in their whitelists before removing them.

The broader crawling infrastructure maintains an extensive network of IP ranges across multiple geographical locations. The AhrefsBot crawler, which operates separately from the Site Audit system, continues to function from multiple locations including Singapore, the United Kingdom, France, Canada, and Germany.

The technical specifications of the IP ranges encompass various network segments. The active ranges include numerous /24 networks, such as 54.36.148.0/24 and 54.36.149.0/24, alongside smaller /26 and /27 subnets. This diverse range of IP blocks allows for robust crawling capabilities across different network architectures.

For organizations using Cloudflare services, the implementation requires special attention due to specific handling of /26 and /27 ranges. In such cases, individual IP addresses must be entered manually to ensure proper whitelisting. The complete list includes hundreds of individual IP addresses across multiple subnets, requiring careful configuration management.

The platform maintains access to these IP addresses through their APIv3, allowing programmatic access to both individual IPs and IP ranges. This programmatic access facilitates automated updates and maintenance of whitelist configurations, particularly beneficial for organizations managing multiple websites or complex network infrastructures.

Technical teams can verify the authenticity of traffic from these IP ranges through the company's support channels, with the organization taking responsibility for all traffic originating from their published IP addresses. This accountability measure provides an additional layer of security and trust in the crawling infrastructure.

Looking at the broader technical implementation, the change represents a strategic shift in how website crawling and auditing services operate. The move to US-based crawling infrastructure potentially impacts various aspects of website analysis, from performance metrics to crawl patterns and data collection methodologies.

The comprehensive nature of these changes necessitates attention from network administrators, security teams, and technical staff responsible for maintaining website accessibility and security configurations. The transition period, spanning from December 23, 2024, to December 31, 2024, provides a window for organizations to update their network configurations accordingly.

Organizations must ensure their technical documentation and network configuration files reflect these changes to maintain optimal functionality with the platform's crawling services. This update affects not only direct users of the service but also any integrated systems or security tools that rely on IP-based filtering or access control mechanisms.