A new analysis of more than 7 billion server log entries has found that OpenAI's automated web crawlers tripled in activity following the launch of GPT-5 in August 2025, even as data from the same period suggests the number of people directly using ChatGPT may be falling. The research, published today by Chris Long, co-founder of SEO and AI-engine optimization consultancy Nectiv, was conducted in partnership with Botify, the enterprise SEO and crawl intelligence platform. It is among the most detailed examinations to date of how OpenAI's three distinct crawlers interact with the open web.

The dataset spans November 2024 through March 2026 and draws on Botify's log file infrastructure, which the company processes on behalf of large enterprise clients across retail and e-commerce, media and publishing, healthcare, software, travel, and marketplaces. According to the analysis, the total dataset exceeds 250 billion log files, of which roughly 7 billion were isolated for this study to focus specifically on events linked to OpenAI's crawlers.

Three crawlers, three purposes

Understanding the findings requires clarity on how OpenAI's crawling infrastructure is organized. According to the research, three bots are in regular operation. ChatGPT-User represents a user-initiated action - when someone instructs ChatGPT to fetch or interact with a page directly, this is the agent that executes the request. GPTBot is OpenAI's general-purpose training crawler, designed to collect data that can be used to improve the foundational knowledge of its language models. OAI-SearchBot is the web search crawler: it operates when ChatGPT performs a search that is not directly triggered by a user, sourcing results for responses that involve real-time web retrieval.

This distinction between crawlers matters because their respective trends have moved in entirely different directions since August 2025 - a divergence that carries different implications for publishers, marketers, and anyone trying to understand how ChatGPT actually reads the web.

PPC Land has covered the evolution of these crawlers in detail. In December 2025, OpenAI revised its crawler documentation, removing training language from the OAI-SearchBot description and expanding the described scope of ChatGPT-User to include Custom GPT requests and GPT Actions. That update clarified that OAI-SearchBot and GPTBot may coordinate - sharing crawl results with each other to avoid duplicate requests when a site has allowed both bots.

The GPT-5 trigger

The central finding of the Botify and Nectiv analysis is that the launch of GPT-5 in August 2025 functioned as a structural inflection point for OpenAI's crawling activity. According to the research, practically overnight, all three of OpenAI's major crawlers registered rapid increases in log events. When the analysis isolates only the automated crawlers - OAI-SearchBot and GPTBot - the difference before and after GPT-5 is described as a "massive" shift. In total, the researchers estimate that OpenAI's crawl of the web has tripled since August 2025.

OAI-SearchBot saw the sharpest acceleration. Comparing activity before and after the GPT-5 launch, the bot registered a 3.5x increase in events. Within Botify's dataset alone, OAI-SearchBot jumped by 2.2 billion events. This growth was not confined to a single industry. According to the analysis, no vertical in Botify's dataset registered negative growth from OAI-SearchBot. The sectors seeing the largest relative increases were healthcare at 740.94% and media and publishing at 701.91%. Marketplaces grew by 215.56%, internet and software by 204.76%, retail and e-commerce by 194.96%, and travel by 29.81%.

GPTBot - the training crawler - also expanded substantially, registering a 2.9x increase since the GPT-5 launch. That translates to a delta of 1.8 billion events when comparing the pre- and post-GPT-5 periods within the same dataset.

The shift aligns with a thesis that circulated in the SEO community at the time of GPT-5's release. According to the Botify blog post, SEO analyst Dan Petrovic had written that GPT-5 signaled a move toward AI systems that are designed to be intelligent rather than knowledgeable - systems that use the web as their live knowledge source rather than relying on static training data. According to the analysis, that interpretation turned out to be accurate.

Search now outpaces training

One of the more precise findings in the study concerns the ratio between searching and training activity. The researchers expressed this as OAI-SearchBot divided by GPTBot and compared the result before and after GPT-5.

Before GPT-5, the ratio stood at 0.95 - meaning OpenAI was spending marginally more time training than searching. After GPT-5, the ratio moved to 1.14, meaning OpenAI now officially spends more time crawling for search purposes than for training. It is a small but directionally significant shift, suggesting the architecture of how ChatGPT retrieves and generates responses has changed.

However, this aggregate picture masks meaningful variation by industry. For media and publishing sites, OAI-SearchBot activity outpaces GPTBot by a 256% relative crawl difference - the highest of any vertical measured. Software and internet sites also lean toward searching. Healthcare and retail and e-commerce move in the opposite direction: healthcare shows a -50% relative difference, and retail and e-commerce shows -33%, indicating GPTBot is relatively more active on those sites. This means a healthcare publisher and a media outlet face materially different optimization challenges when thinking about how OpenAI's systems are interacting with their content.

The ChatGPT-User anomaly

While automated crawling has surged, the story for ChatGPT-User runs in the opposite direction. According to the analysis, comparing December 1, 2025, through March 14, 2026, against the equivalent prior period, ChatGPT-User events dropped by 28%. The decline has been consistent since December 2025 and is described in the research as "dramatic."

Two explanations are offered. The first is straightforward: fewer people may simply be using ChatGPT. The research cites SimilarWeb data showing ChatGPT's traffic share fell from 86.7% within the AI platform category in January 2025 to 64.5% by January 2026. Sistrix data is also cited as showing ChatGPT usage plateauing around late 2025 and then declining.

The second explanation is more structurally interesting. According to the Botify team, a drop in ChatGPT-User events may not indicate fewer conversations at all - it may indicate that OAI-SearchBot is crawling more aggressively and building a more comprehensive index. If OpenAI has assembled a sufficiently fresh HTML web index, the system does not need to fetch pages in real time as often when a user interacts with a page. The analogy used in the research is Gemini, which relies on Google's pre-built index rather than crawling pages on demand when grounding responses. Under this reading, the ChatGPT-User decline could be a sign of improved indexing infrastructure rather than user loss.

The two explanations are not mutually exclusive, and the researchers do not claim to settle the question definitively. What the log file data provides is a measurable proxy: when user-initiated actions decrease and automated crawling increases simultaneously, interpreting the combined signal requires understanding which mechanism is driving each trend.

This ambiguity matters for the marketing community. Publishers and brands that track ChatGPT-User events as a measure of platform engagement may be measuring something that correlates only loosely with the actual volume of conversations that involve their content. PPC Land has noted similar concerns in the context of AI citation research, where blocking crawlers showed limited correlation with actual citation rates.

OpenAI versus Google: still a large gap

The analysis situates OpenAI's crawl expansion within the broader competitive landscape. Even with the tripling of crawl activity, OpenAI's bots represent a fraction of what Google generates. According to the research, in the last month covered by the dataset, Googlebot - both desktop and smartphone - registered 18.2 billion events, while OpenAI's combined crawlers generated 887 million. That places OpenAI at roughly 4% of Google's crawl volume. Bing, for context, registered 5.49 billion events in the same period, meaning OpenAI represents about 14% of Bing's crawl.

The year-over-year comparison, however, shows how quickly the gap is narrowing. In the equivalent 30-day window in 2025, Google crawlers registered 15 billion events while OpenAI registered 207 million, placing ChatGPT at just 1.38% of Google's total crawl. In one year, that share moved from 1.38% to 4%.

PPC Land has tracked this competitive dynamic in related coverage. In January 2026, Cloudflare CEO Matthew Prince disclosed that Googlebot accesses 3.2 times more unique URLs than OpenAI's crawlers and 4.8 times more than Microsoft's, a disparity Prince framed as a structural competitive advantage for Google in AI development. The Botify data adds a time dimension to that picture: the gap exists, but it is measurably smaller than it was twelve months ago.

Earlier data from Cloudflare, reported by PPC Land in December 2025, showed that AI crawlers as a whole accounted for 4.2% of all HTML requests across Cloudflare's global network in 2025, with GPTBot's share of AI crawling traffic rising from 4.7% in July 2024 to 11.7% in July 2025. The Botify study, covering a later period and a different dataset, extends and corroborates that trajectory.

Methodology and scope

The Botify and Nectiv study examined log files generated by Botify's enterprise clients across a range of industries. The overall dataset exceeds 250 billion log entries. The 7 billion-plus entries analyzed for this research were isolated by filtering for events linked to OpenAI's three crawlers. The time window runs from November 2024 through March 2026, with the GPT-5 inflection point identified in August 2025 used as the primary before-and-after comparison point.

Industry segmentation was possible because Botify tags its client data by vertical. This allowed the researchers to compare OAI-SearchBot and GPTBot activity not just in aggregate but across healthcare, media and publishing, marketplaces, software and internet, retail and e-commerce, and travel separately.

The log file methodology itself is worth understanding. A server log records every request made to a website, whether from a human user or an automated bot. Each request generates an entry that includes the user agent string - the identifier that tells the server what kind of software made the request. OpenAI's bots identify themselves using their respective user agent strings: ChatGPT-User, GPTBot, and OAI-SearchBot. Log file analysis at this scale offers something that third-party traffic estimates or crawl simulation tools cannot: a direct observation of what bots actually requested, at what frequency, and across which types of sites.

For the SEO and marketing community, the study reinforces a point that has been building across multiple research publications: server log analysis is the most reliable method for understanding how AI systems interact with a given site, since aggregate statistics and industry benchmarks can obscure significant vertical-level variation.

Implications for publishers and marketers

The findings carry practical weight for different parts of the marketing industry, even if the study stops short of prescriptive recommendations.

For media and publishing sites, the data shows both the highest OAI-SearchBot growth rate among all verticals and the clearest lean toward search over training in the OAI-SearchBot to GPTBot ratio. Publishers in this category are being searched aggressively by OpenAI's systems. That has implications for how frequently their content needs to be fresh and accessible to crawlers. PPC Land has reported that news publishers blocking AI crawlers experienced a post-blocking decline of 23.1% in log-monthly visits according to SimilarWeb data, suggesting the relationship between crawl access and referral traffic is not straightforward but is real.

For healthcare and retail sites, the data points in a different direction. GPTBot is relatively more active on these verticals, indicating that OpenAI is more likely to classify content from those sectors as training material rather than a live search source. Whether that framing changes with future model iterations is unclear from the current data.

The broader picture - that OpenAI is using web search more than it ever has, that its automated bots are at all-time-high activity levels, and that the gap with Google is closing even if it remains large - gives the SEO and GEO (generative engine optimization) communities more concrete data to work with than most prior research. The study does not change the fundamentals of what site owners can do - log analysis, crawl accessibility, fresh content - but it puts measurable numbers behind the trends those communities have been tracking.

Timeline

  • August 7, 2023 - OpenAI announces GPTBot; within two weeks, major sites including Amazon, Quora, The New York Times, and CNN implement blocks. Coverage on PPC Land
  • July 3, 2024 - Cloudflare reports AI bots accessed approximately 39% of the top one million internet properties using its services. Coverage on PPC Land
  • August 3, 2024 - Originality.AI data shows GPTBot blocked by 35.7% of top 1,000 websites, a seven-fold increase from its 5% blocking rate at launch. Coverage on PPC Land
  • August 2025 - GPT-5 launches; OpenAI's automated crawlers - OAI-SearchBot and GPTBot - begin a sustained, rapid increase in activity that the Botify and Nectiv study identifies as the key inflection point.
  • August 29, 2025 - Cloudflare releases data showing Anthropic crawled 38,000 pages per referred visit in July 2025; OpenAI maintained a ratio of 1,091 crawls per referral. GPTBot's share of AI crawling traffic rose from 4.7% to 11.7% year-on-year. Coverage on PPC Land
  • September 2025 - OpenAI commerce capabilities expand; visits from OpenAI to retail websites increase 200% month on month following the rollout, per Botify analysis. Coverage on PPC Land
  • December 2025 - ChatGPT-User log events begin a sustained decline; by March 14, 2026, the drop measures -28% compared to the same period a year earlier.
  • December 9, 2025 - OpenAI revises ChatGPT crawler documentation, removing training language from OAI-SearchBot description and clarifying bot coordination behavior. Coverage on PPC Land
  • December 20, 2025 - Cloudflare's 2025 Year in Review shows AI crawlers accounted for 4.2% of all HTML requests as global internet traffic grew 19%. Coverage on PPC Land
  • January 31, 2026 - Cloudflare CEO Matthew Prince discloses that Googlebot accesses 3.2 times more unique URLs than OpenAI crawlers, framing the disparity as a competitive advantage in AI development. Coverage on PPC Land
  • February 25, 2026 - Anthropic updates crawler documentation, detailing ClaudeBot, Claude-User, and Claude-SearchBot in a structure mirroring OpenAI's tiered crawler model. Coverage on PPC Land
  • March 7, 2026 - Retail Economics, AWS, Botify, and DataDome publish a report showing AI bot traffic at retail sites grew 5.4 times in 2025, with OpenAI generating 1 visit per 198 crawls compared to Google's 1 visit per 6. Coverage on PPC Land
  • March 8, 2026 - Google publishes a new web crawling overview explaining how Googlebot discovers and renders pages. Coverage on PPC Land
  • March 19, 2026 - BuzzStream study of 4 million AI citations finds blocking AI crawlers has little measurable effect on whether a publisher appears in AI-generated responses. Coverage on PPC Land
  • April 23, 2026 - Botify and Nectiv publish analysis of 7 billion-plus OpenAI log file events, finding OpenAI's web crawl has tripled since August 2025 while ChatGPT-User events have dropped 28%.

Summary

Who: Chris Long, co-founder of Nectiv, authored the analysis in collaboration with Botify, an enterprise SEO and crawl intelligence platform. The study draws on log file data from Botify's enterprise client base spanning multiple industries.

What: An analysis of more than 7 billion server log events from OpenAI's three crawlers - ChatGPT-User, GPTBot, and OAI-SearchBot - covering November 2024 through March 2026. Key findings include a tripling of OpenAI's total web crawl since August 2025, a 3.5x increase in OAI-SearchBot, a 2.9x increase in GPTBot, and a -28% decline in ChatGPT-User events since December 2025.

When: The study was published on April 23, 2026, covering a data period from November 2024 through mid-March 2026. The GPT-5 launch in August 2025 serves as the central inflection point in the analysis.

Where: The findings apply globally across all sites accessible to OpenAI's crawlers. Industry-level data covers healthcare, media and publishing, marketplaces, software and internet, retail and e-commerce, and travel verticals.

Why: The research matters because it provides log file-level evidence - rather than estimates or projections - for how OpenAI's infrastructure has changed in response to GPT-5. For the marketing and publishing community, it clarifies which verticals are being crawled for search versus training purposes, quantifies the closing gap between OpenAI and Google in web crawl volume, and introduces a measurable signal that ChatGPT's direct user engagement may be changing even as its automated infrastructure scales.

Share this article
The link has been copied!