Applebot is the web crawler operated by Apple. It fetches pages from the open web and builds the index behind the search results in Spotlight, Siri and Safari, and, since Apple documented the change on June 8, 2026, the data it collects may also be used to train the foundation models behind Apple Intelligence and to supply live context for AI-generated answers. It exists because Apple wanted a search corpus it controlled rather than one rented from Google, Bing or Yahoo, and it has since become the intake pipe for a product line that did not exist when the crawler was built.

For most site operators, Applebot is a name in a server log: less traffic than Googlebot, no console, no query reports, no impression counts. That asymmetry is why its recent changes have gone largely unremarked.

How Applebot fetches and identifies itself

Applebot presents a Safari-shaped user agent string. The general format Apple publishes is Mozilla/5.0 (Device; OS_version) AppleWebKit/WebKit_version (KHTML, like Gecko) Version/Safari_version [Mobile/Mobile_version] Safari/WebKit_version (Applebot/Applebot_version; +http://www.apple.com/go/applebot). Desktop and mobile variants both advertise Applebot/0.1 behind a Safari declaration, and Apple notes that the browser version changes occasionally while the overall shape holds.

Because a user agent string is self-declared, verification runs through two other routes: reverse DNS, where a genuine Applebot address resolves into the applebot.apple.com domain, and a CIDR prefix match against a JSON file Apple publishes at search.developer.apple.com. Both trace back to 17.0.0.0/8, the class A block allocated to Apple, which is how the crawler was identified in the first place. PPC Land's reference on the user agent strings site owners encounter sets out why string matching alone is insufficient across the wider crawler population.

Applebot renders. Apple states the crawler may execute a page inside a browser, so JavaScript, CSS and XHR resources blocked in robots.txt can leave it unable to see the content it came for.

On pacing, one line matters more than its length suggests: Applebot does not follow crawl-delay. Rate is managed internally, adjusting downward when a server slows or returns errors, with crawled content cached to suppress repeat fetches. There is no directive-level lever over frequency.

The robots.txt surface

Applebot honours standard robots.txt directives addressed to it. One fallback rule then shapes behaviour across a large share of the web: if robots instructions say nothing about Applebot but do address Googlebot, Apple's crawler follows the Googlebot rules. Many sites are therefore governing Apple's crawler through directives written for a competitor, without having made any decision about Apple.

At the page level, Applebot supports robots meta tags carrying noindex, nosnippet, nofollow, none and all, either under the generic robots name or an applebot-specific one. Directives can also travel in an X-Robots-Tag HTTP response header, syntax X-Robots-Tag: applebot: nosnippet, the only mechanism available for PDFs, images and other non-HTML resources.

Paywalled material gets its own signal. Applebot supports the schema.org isAccessibleForFree property in page-level JSON-LD; pages marked false stay eligible for search results but are excluded from use as context in AI-generated output. Section-level markup through the hasPart property is not supported, so the unit of control is the whole page.

Three levers, not one

Apple's documentation is unusually direct about the fact that its three controls do not override one another. Standard robots.txt directives govern whether Applebot crawls. Applebot-Extended, a secondary user agent that never fetches a page itself, governs whether already-crawled data may be used to train Apple's general purpose foundation models. The nosnippet tag governs whether content may be used as additional context when models generate output, the example given being broad world knowledge answers in Siri and Search that may carry source links.

Apple states the combined effect plainly: even with Applebot-Extended disallowed and content tagged nosnippet, site instructions may still permit the crawl, and the content remains discoverable through Spotlight, Siri, Safari and other system-wide features. Opting out of training is not opting out of the index. Guidance issued by IAB Australia on July 31, 2026 made the same point, separating control tokens such as Applebot-Extended from the crawlers that honour them.

The Googlebot fallback does not extend to Applebot-Extended. A site that has never named it has, by default, allowed the training use.

Origin and evolution

Applebot surfaced before Apple acknowledged it. Developer Jan Moesen logged hits from an unidentified crawler in the 17.0.0.0/8 block in November 2014. Apple populated a support document in May 2015 describing Applebot as the web crawler for Apple, used by products including Siri and Spotlight Suggestions, and confirming the Googlebot fallback. At the time Siri leaned on Bing and Wolfram Alpha, which is why the disclosure was read as groundwork for something larger.

The second phase arrived in June 2024, when Apple introduced Applebot-Extended. PPC Land documented the mechanics at the time, before Apple Intelligence had shipped. Apple's July 2024 foundation model paper confirmed the corpus, stating that pre-training data included information crawled by Applebot alongside licensed publisher data, decontaminated against 811 common benchmarks. The 2025 technical report described hundreds of billions of pages and crawling weighted by domain-level language identification, topic distribution and URL path heuristics.

The third phase is the June 8, 2026 documentation rewrite, which PPC Land analysed two days later. It moved the training disclosure onto the crawler's own page, introduced the nosnippet rule for AI output, documented X-Robots-Tag support, stated the crawl-delay position and added the paywall property. The same day, Apple confirmed at WWDC26 that Siri AI runs on Google Gemini models.

Why it matters for marketers

Apple ships no equivalent of Search Console. There is no query-level reporting for Spotlight or Siri, no impression data, and no interface for submitting URLs. Server logs are the primary evidence, which makes correct crawler identification the measurement problem in its entirety.

Scale has historically been modest. Cloudflare data covering a two-month window and reported in January 2026 found Googlebot accessing 14.87 times more unique URLs than Applebot, well behind Amazonbot on the same measure. Applebot nonetheless appeared among the higher-volume crawlers in Cloudflare's finding that AI crawlers accounted for 4.2% of HTML requests in 2025, and DoubleVerify's 2025 Global Insights report, released July 22, 2025, counted AppleBot among the AI bots contributing to an 86% rise in General Invalid Traffic during the second half of 2024.

Blocking carries a cost in the other direction. Research from Rutgers Business School and The Wharton School found that news publishers blocking large language model crawlers through robots.txt lost roughly 7% of weekly traffic within six weeks, a figure reported alongside Cloudflare's crawl-to-referral data.

Limitations and disputes

The verification method has a documented gap. Apple's own reverse DNS worked example resolves 17.58.101.179, an address outside every prefix in the published CIDR file. The list is not an exhaustive inventory, and configurations treating it as one will misclassify traffic in both directions.

Adoption of the opt-out has been thin and the measurements disagree. Analysis by Originality AI and Dark Visitors in August 2024 put blocking of Applebot-Extended at roughly 6% to 7% of high-traffic websites, while data journalist Ben Welsh, surveying 1,167 mostly English-language news sites, found 294 blocking it, against 53% blocking OpenAI's crawler and around 43% blocking Google-Extended. The gap reflects different samples rather than a contradiction, and both were read as evidence that few operators had noticed the token.

Retroactivity is unresolved. A robots.txt directive is forward-looking; it does not reach material already crawled into a trained model, and the file carries no legal force. A BuzzStream study published in April 2026 found that blocking AI crawlers rarely stops AI systems citing publisher content, because citations can draw on prior indexes or on search results rather than direct fetches. Compliance is uneven in any case: TollBit measurement published on August 14, 2026 found roughly 15% of identified page-fetching agents in Europe reaching URLs their operators had been told to avoid.

A scheduling conflict now sits in front of Apple. From September 15, 2026, Cloudflare will block crawlers classified as Training and Agent by default on advertising-carrying pages for newly onboarded domains, judging multi-purpose crawlers by their most restrictive behaviour. Applebot is named explicitly alongside Googlebot and BingBot, and Apple's own June 8 disclosure is what places it in that category.

Applebot and adjacent terms

Applebot-Extended is not a crawler. It never requests a page. It is a robots.txt token that determines how data already collected by Applebot may be used for training, and disallowing it has no effect on search indexing.

iTMS is a separate Apple user agent that crawls URLs associated with registered content on Apple Podcasts. Apple states that it does not follow robots.txt, because it is not a general search crawler.

Googlebot is the crawler whose instructions Applebot inherits when no Applebot-specific rules exist. The two remain operationally distinct despite that dependency.

Google-Extended is the closest analogue to Applebot-Extended: a control token, not a fetcher, governing training use by a different operator.

Recent developments

Apple's published address pool expanded sharply in August 2026. Twenty-one new IPv4 prefixes, eighteen /24 blocks and three /28 blocks, added 4,656 addresses to a file that previously listed twelve prefixes covering 2,400, taking the published total to 7,056, an increase of 194%. All the new entries sit inside 17.166.0.0/16 in consecutive runs, consistent with a single allocation rather than accumulated capacity. The change was detected by a monitoring tool built by the London consultancy Merj rather than announced, and reported by Search Engine Roundtable on August 17, 2026. The file carries a creationTime of July 31, 2026, which does not match the detection date.

Address count is not crawl capacity. What the allocation establishes is an intent to distribute requests more widely, the pattern operators follow before parallel fetching increases.

Timeline

  • November 2014: Developer Jan Moesen logs hits from an unidentified crawler operating in Apple's 17.0.0.0/8 address block
  • February 2015: An Apple support document for the crawler appears, initially without content
  • May 2015: Apple populates the document, confirming Applebot as its web crawler for Siri and Spotlight Suggestions and describing the Googlebot fallback
  • August 7, 2023: OpenAI introduces GPTBot, and major publishers begin blocking AI training crawlers within weeks
  • June 2024: Apple introduces Applebot-Extended as a robots.txt opt-out for foundation model training
  • July 2024: Apple's foundation model paper confirms Applebot-crawled web data in the pre-training mixture, decontaminated against 811 benchmarks
  • August 2024: Originality AI and Dark Visitors measure Applebot-Extended blocking at 6% to 7% of high-traffic sites; Ben Welsh finds 294 of 1,167 news sites blocking it
  • July 2025: Apple's updated technical report describes Applebot crawling spanning hundreds of billions of pages
  • January 2026: Cloudflare data shows Googlebot reaching 14.87 times more unique URLs than Applebot
  • June 8, 2026: Apple publishes the rewritten About Applebot documentation covering AI training, nosnippet for AI output, X-Robots-Tag, crawl-delay and paywalled content, and confirms at WWDC26 that Siri AI runs on Gemini models
  • July 31, 2026: The creationTime recorded inside Apple's published Applebot CIDR file
  • August 17, 2026: Twenty-one new prefixes covering 4,656 addresses are detected in that file, taking the published pool to 7,056
  • September 15, 2026: Cloudflare's default block for Training-classified crawlers on advertising pages takes effect for newly onboarded domains

Summary

Who: Apple operates Applebot. The parties affected are website publishers whose content it fetches, SEO practitioners reading server logs, and marketing teams whose bot management and log analysis configurations depend on Apple's published identification data.

What: A web crawler that indexes the open web for Apple's search surfaces and supplies data to Apple's foundation models. It is governed by three independent controls: robots.txt directives for crawl access, the Applebot-Extended token for training use, and the nosnippet meta tag for use as context in AI-generated output. It renders pages in a browser, ignores crawl-delay, falls back to Googlebot instructions when unaddressed, and supports X-Robots-Tag headers and the schema.org isAccessibleForFree property.

When: Logged in November 2014, confirmed by Apple in May 2015, extended with the Applebot-Extended opt-out in June 2024, and substantially redocumented on June 8, 2026. The published address pool expanded on August 17, 2026.

Where: Global. Applebot crawls the open web from Apple's 17.0.0.0/8 address space and feeds Spotlight, Siri, Safari and the foundation models behind Apple Intelligence, Services and Developer Tools.

Why: Apple built Applebot to reduce dependence on third-party search providers for Siri and Spotlight. Its remit has since widened to AI training and retrieval, which places it inside a contested area where publishers, infrastructure providers and regulators are arguing over who controls the web's raw material and on what terms.