GPTBot is the automated crawler OpenAI sends across the public web to gather text that may be used to train its foundation models. It is not a search engine indexer, and it is not the software that fetches a page when someone asks ChatGPT a question. It does one job: it collects material that feeds the training corpus behind the GPT family of models. Its existence gives website operators something they did not have before August 2023, which is a named agent to permit or refuse, and a documented instruction for doing either.

The crawler exists because the legal and commercial pressure on AI training data became impossible to ignore. Before GPTBot, OpenAI had never published a way for a publisher to signal that its pages should stay out of a training set. Datasets were assembled from third-party archives and undisclosed collection, and site owners had no addressable counterparty. GPTBot converted an opaque process into a robots.txt line item.

How GPTBot identifies itself

Three technical artefacts define the crawler. The first is its robots.txt token, which is the string GPTBot. A User-agent: GPTBotblock followed by Disallow: / refuses the whole site; path-level Allow and Disallow rules work as they do for any other well-behaved crawler, so a site can open its blog and close its member area.

The second is the full user-agent header. At launch the string read Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot). OpenAI's crawler documentation, the primary reference for all of this, currently gives GPTBot/1.4 as the example version and warns that the number changes. That detail matters for anyone writing a firewall rule: a regular expression anchored to 1.0 stopped matching some time ago. The documentation also notes that requests for the robots.txt file itself may carry an extra robots.txt marker inside the user-agent string, so that operators whose logs omit paths can still tell a rules fetch from a content fetch.

The third is the published IP range file at openai.com/gptbot.json. As of August 2026 that file lists 21 IPv4 prefixes, most of them in Microsoft Azure allocations, with a creation timestamp of 30 October 2025 and no IPv6 entries at all. The file exists because a user-agent string is a self-declaration that anyone can forge. Verification requires matching the header against the published ranges, and a site that trusts the header alone will admit any scraper that types the word GPTBot into its request.

Where GPTBot sits among OpenAI's other agents

OpenAI documents four separate agents, and conflating them is the single most common error in crawler configuration. GPTBot crawls for training. OAI-SearchBot builds the index behind ChatGPT's search features, and sites opted out of it will not surface in ChatGPT search answers, though they may still appear as navigational links. ChatGPT-User fetches a page when a person or a Custom GPT asks for it. OAI-AdsBot, the newest of the four, visits only pages submitted as advertising landing pages, and OpenAI states the data it collects does not train foundation models.

Each token is independent, which is the point of the design. A publisher can allow OAI-SearchBot for visibility while disallowing GPTBot to withhold training consent. One coupling is documented: where a site permits both, OpenAI reserves the right to use a single crawl for both purposes rather than fetching twice.

Origin and evolution

OpenAI published the GPTBot page on 7 August 2023. The announcement said crawled pages are filtered to strip "sources that require paywall access", sources known to gather personally identifiable information, and text violating company policy. Blocking arrived almost immediately. Data from Originality.AI published on 3 August 2024 found 35.7% of the world's top 1,000 sites refusing GPTBot, against roughly 5% at launch, a sevenfold rise inside twelve months. The same study put CCBot at 22.1%, Google-Extended at 13.6% and ChatGPT-User at 12.7%, which established GPTBot as the most-refused crawler on the web.

The documentation changed materially on 9 December 2025, when OpenAI revised its crawler page. Training language disappeared from the OAI-SearchBot description, ChatGPT-User's described scope widened to cover Custom GPT requests and GPT Actions, and robots.txt compliance language for ChatGPT-User was removed on the reasoning that user-initiated fetches are not automated crawling. Consultant Pieter Serraris identified the change. GPTBot's own definition survived intact, which sharpened rather than blurred its role: it became the only OpenAI agent whose robots.txt directive is unambiguously a training consent signal.

What the crawl volume looks like

Independent measurement, rather than OpenAI disclosure, is the only source of scale figures. Cloudflare data published on 29 August 2025 showed GPTBot's share of AI crawling traffic rising from 4.7% to 11.7% between July 2024 and July 2025, with OpenAI recording 1,091 crawls for every visitor referred back. Anthropic's ratio over the same month was 38,000 to one. Across Cloudflare's network, AI crawlers accounted for 4.2% of all HTML requests in 2025.

An analysis of more than seven billion server log entries by Botify and Nectiv, drawn from a corpus exceeding 250 billion, found that OpenAI's automated crawling tripled after the launch of GPT-5 in August 2025, while ChatGPT-User events fell 28% between December 2025 and mid-March 2026. Training crawling grew as user-triggered fetching shrank. By 2026, HUMAN Security data placed OpenAI's bots collectively at roughly 69% of observed AI-driven traffic.

Why the term matters commercially

For advertising and media operations the crawler is a budget line and a measurement problem before it is a philosophical one. Automated requests inflate server costs, distort session counts and sit in the denominator of every conversion rate a campaign reports. Cloudflare and ETH Zurich argued in research published on 2 April 2026 that crawler behaviour breaks the caching assumptions content delivery networks were built on, degrading performance for human visitors.

The consent question is separate and harder. Refusing GPTBot withholds content from future training runs. It does not withhold it from ChatGPT's answers, because search and retrieval run through different agents and different indexes. That distinction is where most publisher strategy has gone wrong.

Limitations, criticisms and disputes

Four objections recur, and none has been resolved.

The opt-out is prospective only. GPTBot appeared after the bulk of GPT-3.5 and GPT-4 training data had already been collected, a sequencing the Office of the Privacy Commissioner of Canada raised in its ChatGPT findings. No mechanism exists to remove material already absorbed.

Refusal is not protection. Content reaches training sets through Common Crawl and other intermediaries regardless of a site's GPTBot rule. The News/Media Alliance sent Common Crawl a formal demand on 29 April 2026 to stop enabling unauthorised use of publisher content.

Blocking carries measurable cost. Research from Rutgers Business School and The Wharton School found publishers that blocked AI crawlers lost 23.1% of monthly visits in its December 2025 version; the April 2026 revision put the figure near 7% of weekly traffic within six weeks. The two estimates come from the same authors and differ substantially, and the authors attribute the loss mainly to reduced presence inside AI answers rather than lost referral clicks. BuzzStream research published on 19 March 2026 found 70.6% of blocking news sites were still cited anyway.

Compliance is voluntary and unverifiable at the header. Robots.txt carries no enforcement, and the industry now disagrees on whether it governs live agents at all. IAB Australia concluded in July 2026 that it is not a reliable control for that category.

Disambiguation

GPTBot and OAI-SearchBot are frequently treated as one control. They are not. Blocking the former withholds training consent; blocking the latter removes a site from ChatGPT search answers, which is a visibility decision with commercial consequences.

GPTBot and ChatGPT-User differ in who initiates the request. ChatGPT-User acts on a person's instruction, and OpenAI no longer represents it as bound by robots.txt.

GPTBot and CCBot are separate operators entirely. CCBot belongs to the Common Crawl Foundation, a nonprofit whose archives feed many AI developers. Refusing GPTBot leaves CCBot untouched, and Common Crawl's scraper has been documented capturing paywalled articles because it never executes the subscription-check code.

GPTBot and OAI-AdsBot sit on opposite sides of a data boundary. The advertising crawler visits only submitted landing pages, and its output stays inside the advertising pipeline.

Recent developments

Infrastructure has begun deciding the question that robots.txt could not. Cloudflare announced on 1 July 2026 that from 15 September 2026, crawlers it classifies as Training and Agent will be blocked by default on ad-carrying pages for domains newly joining its network, with Search crawlers still permitted. GPTBot falls in the Training category. The same announcement replaced per-crawl charging with payment tied to citations in generated answers.

Enforcement and disclosure are also moving into law. New York's legislature passed the Stealth Crawler Prohibition Act in June 2026, requiring crawlers to identify themselves to news sites with civil penalties up to 15,000 dollars a day. In Europe, the European Data Protection Board adopted Guidelines 03/2026 on web scraping for generative AI on 7 July 2026. TollBit findings reported in August 2026 showed 15% of AI page fetchers in Europe reaching disallowed URLs, a compliance gap that measurement tools such as Microsoft Clarity's June 2026 robots.txt violation reporting now make visible.

Timeline

  • 30 November 2022: ChatGPT launches, built on models trained without any published crawler opt-out.
  • 7 August 2023: OpenAI publishes the GPTBot page and the GPTBot robots.txt token; The Verge, Amazon, Quora, The New York Times and CNN implement blocks within two weeks.
  • June 2024: Cloudflare introduces one-click blocking of AI scrapers for its customers.
  • July 2024: GPTBot accounts for 4.7% of AI crawling traffic on Cloudflare's network.
  • 3 August 2024: Originality.AI reports 35.7% of the top 1,000 websites blocking GPTBot, against 5% at launch.
  • 10 December 2024: Cloudflare launches Robotcop, translating robots.txt rules into firewall rules.
  • July 2025: GPTBot's share of AI crawling traffic reaches 11.7%; Cloudflare opens pay-per-crawl in private beta on 1 July.
  • 7 August 2025: GPT-5 launches; OpenAI's automated crawling begins a sustained increase.
  • 29 August 2025: Cloudflare publishes crawl-to-referral ratios, placing OpenAI at 1,091 crawls per referral.
  • 30 October 2025: Timestamp on the current version of the published GPTBot IP range file.
  • 9 December 2025: OpenAI revises its crawler documentation, removing robots.txt compliance language for ChatGPT-User.
  • 31 December 2025: Rutgers and Wharton researchers post their study of publisher blocking outcomes.
  • 21 April 2026: Revised version of the same study lowers the estimated traffic cost to roughly 7% weekly.
  • 26 April 2026: OAI-AdsBot documented as OpenAI's fourth crawler.
  • 29 April 2026: News/Media Alliance sends its demand letter to Common Crawl.
  • 1 July 2026: Cloudflare sets 15 September 2026 as the start of default blocking for Training crawlers on ad-carrying pages.
  • 7 July 2026: EDPB adopts Guidelines 03/2026 on web scraping for generative AI.

Summary

Who: OpenAI operates GPTBot. The parties affected are website operators, publishers, advertisers whose landing pages sit on the open web, and the infrastructure companies (Cloudflare, TollBit, Microsoft Clarity) that now mediate crawler access. Regulators in New York, the European Economic Area and Canada have all addressed the crawler or the practice behind it.

What: An automated web crawler that collects publicly reachable content for possible use in training OpenAI's foundation models, identified by the robots.txt token GPTBot, a versioned user-agent string currently documented at 1.4, and 21 published IPv4 prefixes.

When: Launched 7 August 2023. Documentation last materially revised on 9 December 2025. Cloudflare's default-blocking policy for training crawlers takes effect 15 September 2026.

Where: Across the public web, with the highest measured crawl intensity on media, publishing and retail domains, and the highest refusal rates among large news publishers.

Why: GPTBot is the only addressable point at which a site can signal that its content should not train OpenAI's models. That signal is voluntary, prospective, unenforceable at the header level, and demonstrably costly to exercise, which is why the argument over it has moved from robots.txt into infrastructure defaults, licensing protocols and legislation.