ClaudeBot is the automated program Anthropic sends across the public web to collect page content that may be used to train its Claude family of language models. It identifies itself in server logs through a user agent string, it reads the robots.txt file at the root of a domain before fetching anything else, and it stops crawling a site that tells it to stop. Anthropic describes its purpose in a single sentence: the bot collects web content that could potentially contribute to model training.

The crawler exists because training a large language model requires text at a scale no licensing deal can supply on its own, and because the web is the only corpus large enough. It matters to publishers and marketers for the opposite reason. Every page ClaudeBot fetches is a page taken without payment, served at a bandwidth cost the site owner absorbs, and the only lever site owners have is a plain-text file that no law requires any crawler to obey.

How the crawler works

The sequence is conventional. Before requesting pages on a hostname, the crawler fetches /robots.txt and parses it for a group matching its token. The token is the literal string ClaudeBot. A directive addressed to a browser string, an IP range, or one of Anthropic's older agent names will not match.

Server logs record the request through a user agent header. The commonly documented form is Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com). Anthropic supplies an email address as the operator contact rather than a documentation URL, a departure from OpenAI's convention of linking a page for each bot. PPC Land noted in June 2026 that ClaudeBot also omits the AppleWebKit token that most crawlers inherited from browser strings, though third-party crawler databases record log entries containing it.

Two directives govern behaviour. A site-wide block takes the standard form, and Anthropic states it must be repeated on every subdomain to be excluded:

User-agent: ClaudeBot Disallow: /

Rate is handled through Crawl-delay, an extension that never entered the formal standard. Anthropic supports it and publishes an example using a one-second interval. Robots Exclusion Protocol itself dates to 1994 as an informal convention and was only codified as RFC 9309 in September 2022, which is the substance of the complaint made repeatedly by publishers: a voluntary file written for search indexing now carries the weight of a licensing decision.

Verification is the weaker half of the mechanism. A user agent string is a header any client can set, so a request claiming to be ClaudeBot proves nothing by itself. Anthropic publishes an address list at claude.com/crawling/bots.json, which as of a file generated on August 13, 2026 contained 23 IPv4 prefixes: one /22, three /28 blocks, and nineteen single addresses. The ranges resolve into Google Cloud and Microsoft Azure address space rather than networks Anthropic operates directly, which is the reason the company warns that blocking addresses is unreliable and can prevent the crawler from reading robots.txt at all. The list carries no per-bot labels, so it cannot separate training traffic from the other two agents.

Three bots, three consequences

Anthropic operates ClaudeBot alongside two other agents, and the distinction is the operationally important one. Claude-User fetches a page when a person using Claude asks a question that requires live web content. Claude-SearchBot crawls to improve the quality of search results inside the product.

Anthropic documented the separation and the cost of blocking each on February 25, 2026, after the change was spotted on February 20 by Pedro Dias and reported by Barry Schwartz at Search Engine Roundtable. Restricting ClaudeBot signals that a site's future material should be excluded from training datasets. Restricting Claude-User stops the system retrieving content in response to a live query, which the documentation says may reduce visibility for user-directed web search. Restricting Claude-SearchBot prevents indexing for search.

One point separates Anthropic from its peers. All three agents are stated to honour robots.txt. OpenAI removed compliance language from its ChatGPT-User description on December 9, 2025, arguing that user-initiated fetches are not automated crawling, and Google formalised the same category on March 20, 2026 by adding Google-Agent to its list of user-triggered fetchers. IAB Australia sorted every crawler into one of four verdicts on July 31, 2026 and concluded that robots.txt is not a reliable control for live agents at all.

Origin and evolution

ClaudeBot first appears in robots.txt files in December 2023, on 2,382 sites according to HTTP Archive analysis by Paul Calvano, rising to roughly 30,000 within four months and passing 100,000 by May 2024. It replaced two earlier agents, anthropic-ai and Claude-Web, both since deprecated. Anthropic told 404 Media in July 2024 that it had configured ClaudeBot to respect directives previously set for those names, a statement that mattered because major publishers including Reuters and the Conde Nast titles were at that point blocking only the retired strings.

The bot became a public issue in mid-2024. In April, a Linux Mint forum outage was attributed to its crawling. On July 24, iFixit chief executive Kyle Wiens posted that ClaudeBot had hit the repair site roughly one million times in 24 hours. Freelancer chief executive Matt Barrie told The Information his site absorbed 3.5 million requests in four hours, calling it the most aggressive scraper by a factor of about five over the next-largest. Wiens later told The Register that crawling stopped once a disallow directive was added, though the file itself was then fetched every thirty minutes.

Cloudflare quantified the imbalance in August 2025 through the crawl-to-refer ratio, the count of pages fetched for each visitor sent back. Anthropic recorded 286,930 crawls per referral in January 2025, improving to 38,000 by July. In the same dataset ClaudeBot's share of AI crawling traffic rose from about 6% to roughly 10%.

Why the term matters for marketers

The decision ClaudeBot forces is not a technical one. Blocking it removes a site's future pages from training data, which over time reduces the likelihood that a model reproduces a brand's positioning unprompted. It does not remove the site from Claude's live answers, because those run through the other two agents. A single line in robots.txt therefore separates a data rights position from an audience reach position, and getting them confused is the common error.

The cost of refusal has been measured. Research from Rutgers Business School and The Wharton School, revised on April 21, 2026, found news publishers blocking model crawlers through robots.txt lost roughly 7% of weekly traffic within six weeks, with a figure of 23.1% of monthly visits reported elsewhere in the same body of work. Neither number reflects compensation, because the protocol carries none.

Limitations and disputes

Compliance is asserted, not proven. TollBit measurement reported in August 2026 found 15% of AI page fetchers in Europe reaching disallowed URLs, and Microsoft Clarity began flagging robots.txt violations inside Bot Analytics in June 2026 precisely because publishers had no independent evidence either way.

Anthropic's own record is contested. Reddit sued the company on June 4, 2025 in San Francisco Superior Court, alleging that bots accessed the platform more than 100,000 times after Anthropic said it had stopped. Separately, Anthropic agreed to a settlement of at least 1.5 billion dollars on September 5, 2025 covering roughly 500,000 works at about 3,000 dollars each, after Judge William Alsup found training itself transformative but rejected the piracy defence on June 23, 2025. That case concerned downloaded books rather than crawled pages, a distinction routinely lost in summary.

Measurement is disputed too. Cloudflare's vertical breakdown put Anthropic near 50,000:1 on the default view and 8,800:1 for news and publications in August 2025. Third-party readings of Cloudflare Radar for mid-2026 place ClaudeBot in the low thousands per referral, with figures between roughly 1,900 and 11,000 circulating for overlapping periods. The spread reflects operator-level and user-agent-level aggregation being compared as though identical.

ClaudeBot and adjacent terms

Claude-User is a real-time fetcher, not a crawler. It runs when a person asks a question, retrieves specific pages, and can send a referral. Claude-SearchBot indexes for search quality. A site owner who blocked Claude-SearchBot at the firewallreported 420,680 page requests in one week against 12 human visitors from Claude-User.

anthropic-ai and Claude-Web are retired tokens still copied into robots.txt files from stale block lists. Directives naming them have no effect on current traffic.

GPTBot is OpenAI's equivalent training crawler. Google-Extended is not a crawler at all but a policy token that withholds content from Gemini training without affecting Googlebot's search indexing, which is why blocking it costs nothing in search visibility.

Recent developments

Anthropic's crawler page was updated again on April 7, 2026, adding the bots.json address list that the February documentation had explicitly said did not exist. Verification by address is now possible, though the list remains unlabelled by bot.

Cloudflare published Bot Preference Sync on August 21, 2026, generating robots.txt from dashboard policy and attaching four disclosure conditions for operators whose bots perform both search and training. Anthropic's separation of its three agents satisfies the first condition. The third, URL-level reporting on which pages were made available for training, has no equivalent at any major operator including Anthropic. From September 15, 2026, Cloudflare blocks training and agent crawlers by default on advertising-carrying pages for domains newly joining its network.

Timeline

  • 1994: The Robots Exclusion Protocol appears as an informal convention for controlling crawler access
  • September 2022: The protocol is codified as RFC 9309, nearly three decades after adoption
  • December 2023: ClaudeBot first appears in robots.txt files, on 2,382 sites in HTTP Archive data
  • April 2024: A Linux Mint forum outage is attributed to ClaudeBot crawling activity
  • May 2024: References to ClaudeBot pass 100,000 sites in robots.txt files
  • July 24, 2024: iFixit reports roughly one million ClaudeBot requests in 24 hours
  • July 2024: Freelancer reports 3.5 million requests in four hours; Anthropic confirms anthropic-ai and Claude-Web are retired
  • January 2025: Cloudflare records Anthropic at 286,930 crawls per referral
  • June 4, 2025: Reddit sues Anthropic in San Francisco Superior Court over alleged unauthorized data collection
  • June 23, 2025: Judge William Alsup finds AI training transformative but rejects the piracy defence
  • July 2025: Anthropic's crawl-to-refer ratio improves to 38,000 per referral
  • September 5, 2025: Anthropic agrees to a settlement of at least 1.5 billion dollars over pirated books
  • December 9, 2025: OpenAI removes robots.txt compliance language from its ChatGPT-User description
  • February 20, 2026: Pedro Dias spots Anthropic's revised crawler documentation
  • February 25, 2026: The three-bot breakdown is reported, with all three agents committed to robots.txt
  • March 20, 2026: Google adds Google-Agent to its list of user-triggered fetchers
  • April 7, 2026: Anthropic's crawler article is updated, adding the bots.json address list
  • April 21, 2026: Rutgers and Wharton researchers post revised findings on traffic loss from blocking
  • July 31, 2026: IAB Australia concludes robots.txt is not a reliable control for live agents
  • August 13, 2026: The published bots.json file lists 23 IPv4 prefixes
  • August 21, 2026: Cloudflare publishes Bot Preference Sync and four transparency conditions
  • September 15, 2026: Cloudflare begins default blocking of training crawlers on ad-carrying pages for new domains

Summary

Who. Anthropic PBC operates ClaudeBot. Site owners, publishers and technical SEO teams decide whether to allow it, and infrastructure providers including Cloudflare and Microsoft supply the tooling used to observe and enforce those decisions.

What. An automated web crawler that collects public page content which may contribute to training Anthropic's Claude models. It is one of three Anthropic agents, distinguished from Claude-User, which fetches pages for live queries, and Claude-SearchBot, which indexes for search quality.

When. The identifier first appeared in robots.txt files in December 2023, drew public complaints from iFixit and Freelancer in July 2024, and gained formal per-bot documentation on February 25, 2026, with an address list added on April 7, 2026.

Where. Across the public web, requesting pages over HTTP from addresses published at claude.com/crawling/bots.json, and controlled through the robots.txt file at the root of each hostname and subdomain.

Why. Language models require training text at a scale no licensing arrangement supplies, and the open web is the available source. The crawler matters commercially because it consumes publisher infrastructure while returning far fewer referrals than search crawlers, making the choice to allow or block it a decision about data rights and audience reach at the same time.