Cloudflare on July 1, 2026 opened a new dashboard to Bot Management customers that breaks down, company by company, how many times an AI crawler visits a website compared with how many human readers that same company sends back through a link. The tool, called Attribution Business Insights, arrived as part of the company's second annual Content Independence Day, an event Cloudflare has used each July to introduce changes in how its network treats automated traffic from artificial intelligence systems.

The dashboard does not block anything on its own. It sits alongside the security rules that already exist inside Bot Management and gives a new audience, described in the announcement as business decision-makers rather than security engineers, a single screen showing which companies are crawling a site, what those companies do with the content, and how much traffic comes back in return.

What the dashboard measures

According to the Cloudflare Blog post written by Jin-Hee Lee and Oliver Payne, the central problem the dashboard addresses is a breakdown in what the post calls the historical bargain of the internet: search engines crawled a page, then sent a visitor back, and that exchange funded advertising, subscriptions and affiliate revenue for the site that made the content available. The post describes traditional search crawlers scanning a site's content only a small number of times for each referral they sent, a ratio that kept the underlying economics workable for publishers.

That balance no longer holds for a large share of AI-related traffic, the company argues. The post describes ratios collected around the time of the first Content Independence Day in 2025 ranging from 118 crawls for every referral up to nearly 50,000 crawls for every referral. Put differently, a crawler operating at the high end of that range could visit the same page tens of thousands of times before a single reader clicked through from that company's product. Cloudflare frames the shift as part of a broader move from an internet organized around search engine optimization toward one increasingly shaped by answer engine optimization and, more recently, generative engine optimization, as chatbots and AI search products synthesize responses directly from crawled material rather than sending users to the source.

The dashboard itself surfaces four categories of information, all viewable across three time windows: the last 24 hours, the last 7 days, or the last 30 days. The first is overall bot traffic to content pages, shown against total human traffic, so a site owner can see what share of requests come from any kind of automated system, not only AI-related ones. The second is the crawl-to-referral ratio itself, available both as a single site-wide figure and broken out per bot operator, meaning per company that owns one or more crawlers. The third is a ranked list of the bots generating the most volume, along with each bot's country of origin, the bandwidth it consumes, and whether the site is currently blocking, partially blocking, or allowing it. The fourth is a classification layer that sorts crawlers by what they are actually doing rather than lumping every automated visitor under a single generic label.

The three-way classification

Cloudflare's documentation for the dashboard, published alongside the announcement, describes the classification scheme as Training, Search, and Agent. A Training classification means the crawler is gathering material to improve a future version of a language model. A Search classification covers bots that refresh databases used for retrieval-augmented generation, a technique where a system pulls in current web content to ground its answers rather than relying solely on what it learned during training. An Agent classification applies to crawlers acting on behalf of a specific user during a live interaction, fetching a page because someone asked a question that required checking that page in the moment.

Cloudflare's supporting documentation defines several of the dashboard's core metrics precisely. The crawl-to-referral ratio, whether shown per operator or site-wide, is calculated as the number of crawls a company's bots send against the number of visitors who arrive through a referral link from that same company, with referrals tracked through UTM parameters. The documentation also specifies that a company is labeled "AI" on the dashboard if it operates at least one bot with an AI use case, and that companies appear as Blocked, Allowed, or Partially Blocked, with the partial label applying whenever some of a company's bots are blocked while at least one remains allowed.

Availability instructions differ slightly between Cloudflare's two published sources. The blog post announcing the dashboard states it is available to all Cloudflare Bot Management customers, while Cloudflare's supporting documentation specifies availability for Bot Management Enterprise customers, reachable either through a direct link or through the zone-level Analytics menu inside the Cloudflare dashboard. The documentation states plainly that the tool is meant for visibility rather than control, and that publishers who want to act on what they see still need to configure Security rules or Cloudflare's separate AI bot mitigation settings, since the dashboard itself introduces no new enforcement mechanism.

Why Cloudflare built a business-facing view

The blog post argues that the crawl-to-referral imbalance creates what it calls a double cost for publishers. First, sites lose the referral traffic, advertising impressions, and direct audience relationships that historically funded content production. Second, they still have to pay the infrastructure costs of serving requests to bots that, in the case of Training crawlers particularly, return no commercial value at all. Whether every reader agrees with that framing is a separate question, but it is the stated rationale Cloudflare gives for building a dashboard aimed at people making decisions about vendor relationships and licensing, not only the security staff who configure firewall rules.

The post frames the new audience explicitly, stating that the dashboard invites "business decision-makers into this dashboard, acknowledging that conversations around AI traffic have a wider set of stakeholders than only security-specialized users." That framing matters because it signals Cloudflare's own view of where the AI traffic conversation is heading: away from a purely technical question of what to block, and toward a commercial one about which companies are worth a licensing conversation and which are not.

The post also offers a specific illustration of how a publisher might use the per-operator view in a negotiation, describing a scenario in which a site owner could tell one company its crawl volume runs twenty times higher than another company that is already paying for content access, or revisit licensing terms with a third company based on its recent crawling activity. Those examples are illustrative rather than drawn from a named customer case, and they point to the dashboard's intended use: arming a business conversation with figures rather than estimates.

The wider Content Independence Day announcements

The Attribution Business Insights launch was one of several changes Cloudflare introduced on July 1, 2026, all tied to the same Content Independence Day framing the company first used a year earlier. On the same day, Cloudflare also announced a shift in its payment model for AI crawlers, moving away from charging per individual crawl and toward a structure that pays publishers based on whether their content was actually used to generate an answer, a change the company tied to internal data showing that more than half of crawl traffic from bots it considers legitimate re-fetches pages that have not changed since the previous visit.

A related July 1 policy update, covered separately, set new default rules for crawlers on monetized pages, stating that starting September 15, 2026, Training and Agent crawlers will be blocked by default on pages carrying advertising for domains newly onboarding to Cloudflare's network, while Search crawlers remain allowed by default under the same policy. Attribution Business Insights does not itself implement that new default; it is the visibility layer, while the default-blocking policy and the payment restructuring are separate mechanisms announced the same day.

Taken together, the three announcements describe a company trying to move past a strategy built entirely around the HTTP 402 "Payment Required" status code, which Cloudflare first activated for AI crawler monetization on July 1, 2025 through its Pay Per Crawl service, then expanded on August 28, 2025 with customizable payment and licensing messages inside those 402 responses. Attribution Business Insights adds a fourth layer to that same effort: rather than charging for access or classifying crawlers by category alone, it gives a business user the raw comparative numbers first, on the theory that a licensing or blocking decision should follow data rather than precede it.

How this fits the broader crawl-to-referral record

The specific ratios Cloudflare cites in its Attribution Business Insights announcement, 118 crawls per referral at the low end and nearly 50,000 at the high end, sit within a longer pattern the company has published data on repeatedly over roughly the past two years. Cloudflare's own historical figures, tracked across several disclosures, showed one operator's ratio reaching 286,930 crawls per referral in January 2025 before falling to 38,000 crawls per referral by July 2025, alongside a separate operator holding steadier at 1,091 crawls per referral over the same stretch. Whether the newest dashboard's per-operator figures will show ratios continuing to fall, as that earlier data suggested, or whether the trend will reverse as crawling volume keeps rising, is not something the July 1 announcement itself addresses; it is a question the dashboard is designed to let publishers track going forward rather than one Cloudflare answers definitively today.

That broader context also includes research on what happens when publishers respond to high crawl ratios by blocking crawlers outright. A study from Rutgers Business School and The Wharton School, most recently revised on April 21, 2026, found that news publishers who blocked large language model crawlers through robots.txt lost roughly 7 percent of weekly website traffic within six weeks of implementing the block, with the decline visible in human browsing panel data rather than only in automated bot metrics. That finding complicates a simple read of Cloudflare's own framing: high crawl ratios clearly strain infrastructure and displace referral revenue, yet the same research suggests that severing access entirely does not cleanly restore the traffic or protection publishers might expect, since robots.txt remains a voluntary standard that a non-compliant crawler can simply ignore.

Separately, Cloudflare and researchers at ETH Zurich published joint findings in April 2026 arguing that AI crawler traffic breaks standard assumptions built into web caching systems, in part because more than 90 percent of the pages processed monthly by large-scale crawling operations such as Common Crawl are unique by content, defeating the logic that caches were designed around: that popular pages get requested again and again. That research adds a technical dimension to the same underlying complaint Attribution Business Insights is meant to surface commercially, namely that AI crawler behavior does not resemble the search-engine crawling patterns that current web infrastructure, and current publisher business models, were built to accommodate.

What the dashboard does not do

Cloudflare's own documentation is explicit that Attribution Business Insights introduces no new control mechanism. The dashboard is described as a visibility layer sitting on top of existing tools, and any action a publisher decides to take after reviewing the data, whether blocking a specific operator, allowing it, or leaving current settings unchanged, still happens through Security rules elsewhere in the Bot Management product. The distinction separates the dashboard from Cloudflare's earlier pay-per-crawl mechanism, which combined visibility and a payment-collecting action inside a single feature. Here, the two functions remain split: Attribution Business Insights answers the question of what is happening, and existing rule engines answer the separate question of what to do about it.

Cloudflare also disclosed that a further iteration is already under development with what the post describes as close publishing partners, intended to break down crawler activity at the level of an individual article rather than only at the level of an entire site. The current dashboard's most granular view groups activity by bot operator across a whole domain; the planned addition would let a publisher see which specific pieces of content draw the heaviest AI crawler interest, and from which companies. Cloudflare did not attach a launch date to that planned feature in the July 1 announcement.

Timeline

  • July 1, 2025: Cloudflare launches Pay Per Crawl in private beta, introducing HTTP 402 Payment Required responses for AI crawler monetization and declaring the first Content Independence Day.
  • August 28-29, 2025: Cloudflare expands AI Crawl Control with customizable 402 responses and publishes data showing one operator's crawler accessing 38,000 pages for every referred visit in July 2025.
  • April 2, 2026: Cloudflare and ETH Zurich publish joint research documenting how AI crawler traffic breaks standard web caching assumptions.
  • April 21, 2026: The most recent revision of the Rutgers-Wharton study on AI crawler blocking and publisher traffic loss is posted to SSRN.
  • July 1, 2026: Cloudflare launches the Attribution Business Insights dashboard for Bot Management customers, alongside a separate shift in its AI crawler payment model and a new default-blocking policy for monetized pages, marking the company's second annual Content Independence Day.

Summary

Who: Cloudflare, the web infrastructure company, built the Attribution Business Insights dashboard, with the announcement authored by Jin-Hee Lee and Oliver Payne. The tool is aimed at Bot Management customers, particularly business decision-makers and publishers rather than security specialists alone.

What: A new analytics dashboard that shows site-wide and per-company crawl-to-referral ratios, a ranked breakdown of the bots generating the most traffic, and a three-part classification of crawlers as Training, Search, or Agent, all viewable across 24-hour, 7-day, or 30-day windows.

When: Cloudflare announced and launched the dashboard on July 1, 2026.

Where: The dashboard is accessible globally to Bot Management customers through the Cloudflare dashboard's Analytics section, applying to any website using Cloudflare's Bot Management product. Cloudflare's own sources differ slightly on whether this covers all Bot Management customers or specifically the Enterprise tier, as detailed in the article above.

Why: Cloudflare states that AI crawlers have broken the historical exchange between crawling and referral traffic that funded advertising and subscription revenue for website owners, with ratios reaching as high as nearly 50,000 crawls for every referral, and that publishers need company-level data to decide which AI crawlers merit blocking, allowing, or a licensing conversation.