Cloudflare today announced a shift in how it plans to compensate website owners for content used by artificial intelligence search tools, moving from a model that charges for each individual crawl toward one that pays based on whether that content was actually used to answer a question. The announcement, made in a post by Matthew Conroy on the Cloudflare Blog, arrives on the company's second annual Content Independence Day, a self-designated date the infrastructure provider uses each July 1 to mark changes to how it handles automated traffic from AI systems.

The core claim in today's post rests on a data point Cloudflare has been building toward for months: more than half of the crawl traffic generated by bots it classifies as legitimate goes toward re-fetching pages that have not changed since the last visit. According to Cloudflare, that share is likely to grow as crawl volumes increase, meaning a rising proportion of the requests hitting publisher servers produce no new information for anyone. The company frames this as waste on both sides of the transaction: AI companies spend compute cycles retrieving stale content, and publishers pay hosting costs to serve it.

A research program built on freshness signals

The centerpiece of today's announcement is a research initiative Cloudflare is calling an effort to make AI search smarter. The company says it sits behind more than 20% of websites on its global network, a position it argues gives it visibility that no single publisher or AI company can replicate on its own. Through the program, Cloudflare intends to combine two data sources: signals that customers voluntarily share about when their content has genuinely changed, and Cloudflare's own observation of traffic patterns, covering both human visitors and automated bots.

The stated goal is to measure two separate outcomes. The first is whether these combined signals help AI answer engines surface content that is fresher and of higher quality. The second is how much unnecessary crawling the signals eliminate. According to the post, a signal that simply confirms nothing has changed on a page lets a crawler skip that page entirely, saving compute for the AI company and saving bandwidth costs for the site owner. Cloudflare describes the program as neutral by design, stating that it intends to make the underlying capability available to any answer engine willing to participate, and that it plans to publish results including benefits such as improved content discoverability and reduced server load.

The company was explicit about scope limits. According to the post, the program is restricted to search functions, no content is being shared through it, and none of the data involved is used to train foundation models. Cloudflare said it plans to make the capability broadly available later in 2026, though no specific date accompanied that projection.

Why re-crawling became a cost problem

The freshness argument did not emerge in isolation. Cloudflare has spent much of the past year documenting how automated traffic behaves differently from human visitors, and how that difference strains infrastructure built for an earlier internet. Research the company published with ETH Zurich in April 2026 found that AI crawlers were breaking assumptions built into standard web caching, because more than 90% of the pages processed by large-scale crawlers like Common Crawl are unique by content, a pattern that defeats caching systems designed around the expectation that popular pages get requested again and again.

That research also quantified how large the automated share of web traffic has become. By early June 2026, bots accounted for 57.4% of all web traffic to HTML content tracked across Cloudflare's network, with human visitors making up the remaining 42.6%. Within that bot majority, training-related crawlers represented 50.6% of total traffic, while search-related bots accounted for just 10.7%. The distinction matters for today's announcement because a search bot creates at least a theoretical path toward a human clicking through from a result page, while a training bot extracts content with no comparable payoff for the publisher.

From Pay Per Crawl to Pay Per Use

A year ago, on the first Content Independence Day, Cloudflare launched Pay Per Crawl in private beta, a system that let content owners charge AI crawlers for access using HTTP 402 Payment Required responses, a largely dormant web standard originally reserved for future digital payment systems. Publishers configured one of three settings for each crawler: allow free access, charge a set price, or block access outright. Cloudflare served as merchant of record, handling the billing when a crawler made an authenticated request carrying payment intent.

The company expanded that framework in August 2025, when it rolled out customizable HTTP 402 responses allowing publishers to embed specific pricing and licensing instructions directly inside the payment-required messages sent back to crawlers. That expansion coincided with Cloudflare's broader AI Week programming and came alongside data showing the imbalance the pay-per-crawl model was designed to address: at the time, Anthropic's crawler accessed 38,000 pages for every one visit it referred back to a publisher, while OpenAI's ratio stood at 1,091 crawls per referral.

Today's post reframes that entire approach as a first step rather than an endpoint. According to Cloudflare, crawling is a poor proxy for value because a single page might be crawled once and then cited in thousands of AI-generated answers, or crawled repeatedly and never cited at all. The stated aim is to move the unit of compensation from the crawl itself to the outcome the crawl produces - specifically, whether the content ends up inside an answer a user actually reads.

Two commercial experiments, two different mechanics

Cloudflare is running the shift as a pair of experiments with named partners rather than as a single unified product. The first involves Ceramic.ai, whose founder and chief executive, Anna Patterson, described the arrangement in a quoted statement included in the announcement. "To scale the future of AI search, we need a partner with massive reach and a shared commitment to transparency and fair compensation," Patterson said. "Cloudflare allows us to easily and programmatically scale our operations. By bringing our pay-per-query model to their network, we ensure millions of content owners can seamlessly opt in to be compensated every single time their content appears in our search results."

According to Cloudflare, Ceramic has built what the post calls a pay-per-query model, under which publishers who opt in are paid when their content appears in Ceramic's search results, rather than each time a crawler happens to fetch a page. Participating publishers also gain access to new reporting Cloudflare describes as useful for what it terms answer engine optimization, including the specific queries that led to their content appearing in results, the exact webpage and snippet shown, and the average ranking position of that content within Ceramic's output.

The second experiment involves You.com, structured differently. According to the post, You.com allows AI agents to pay on demand for a specific piece of premium content at the moment it is needed, without requiring any upfront commitment from either side. Cloudflare characterized both arrangements as early examples of a broader category of emerging payment models - including approaches the post refers to as pay per query and pay per result - and said its infrastructure is designed to support the range of models AI companies are testing, rather than mandating one specific mechanism.

Context: a year of documented traffic decline

Today's announcement lands against a backdrop of accumulating research on how AI-generated search summaries affect the traffic publishers depend on. Cloudflare's post cites a 2025 Pew Research Center study finding that when Google displays an AI summary, users clicked a traditional search result link just 8% of the time, roughly half the rate observed when no summary appeared, and clicked a link inside the AI summary itself in only 1% of cases. That figure aligns closely with what Pew's own July 2025 publication documented: an 8% click rate with AI summaries present versus 15% without them, based on browsing data collected from 900 U.S. adults during March 2025.

Google publicly disputed those findings at the time, calling Pew's methodology "flawed" and its query set "skewed."Subsequent research from other organizations has produced figures in a similar range, though not identical. Ahrefs, which examined 300,000 keywords, found in April 2025 that AI Overviews correlated with a 34.5% reduction in clicks to the top-ranking organic result; a follow-up Ahrefs analysis published in February 2026 put that figure at 58%, nearly double the earlier measurement. Seer Interactive, analyzing 42 client organizations, reported in November 2025 that organic click-through rates on AI Overview queries had fallen 61% between June 2024 and September 2025, while paid click-through rates over the same window dropped 68%, from 19.7% to 6.34%.

The pattern extends beyond Google's own product. Cloudflare's account of more than 50% of good-bot crawl traffic going toward unchanged pages fits alongside earlier company data showing training-related crawling reaching nearly 80% of all AI bot activity by September 2025, up from 72% a year earlier, while search-related crawling had fallen to 18% of the total and user-initiated actions accounted for just 2%.

Answer engine optimization as an emerging discipline

Cloudflare's decision to build reporting tools around query-level visibility places today's announcement within a wider shift the marketing industry has been navigating for roughly a year. Answer engine optimization, often abbreviated AEO, describes efforts to track and influence how brands and publishers appear inside AI-generated responses rather than traditional search rankings. HubSpot launched a dedicated AEO product in April 2026 priced at 50 dollars per month, disclosing at the time that organic traffic for its own customers had fallen 27% year over year. The company followed that in May 2026 with a free public dashboard called AEO Sensor, tracking industry-level shifts in how ChatGPT, Gemini, and Perplexity treat brands without requiring a login or subscription.

Microsoft Advertising published its own playbook addressing the discipline in January 2026, framing answer engine optimization and generative engine optimization as successors to conventional search engine optimization for retailers navigating agentic commerce. Not every voice in the industry has welcomed the proliferation of new terminology. SparkToro co-founder Rand Fishkin argued in May 2025 that the marketing field would be better served retaining the existing SEO acronym with an expanded definition, rather than adopting AEO, GEO, and similar labels; Google's own Search Central guidance, published in May 2026, took a related position, describing AI search optimization as still falling under the umbrella of SEO, a framing that drew pushback from iPullRank founder Mike King, who argued the framing serves Google's platform interests over publishers'.

Cloudflare's reporting tools, tied specifically to the Ceramic partnership, sit adjacent to this trend without adopting its terminology directly. The company's own phrase, answer engine optimization, appears once in the post to describe the category of reporting it plans to offer, without further elaboration on how that reporting might integrate with the broader AEO tooling already sold by companies including HubSpot, Microsoft, and Typeface.

What Cloudflare is not claiming

The post is notably cautious about outcomes. Cloudflare describes the entire effort as an experiment and states plainly that there is a lot to learn, including how the approach holds up at the scale of the internet. No pricing structure was disclosed for either the Ceramic or You.com arrangements, and no timeline beyond "later this year" was given for broader availability of the freshness-signal research program.

The framing throughout the announcement positions Cloudflare as an infrastructure layer rather than as a party setting prices or negotiating terms. The company states that its role is to provide the plumbing that allows organizations to bring their own payment models and scale them across the publishers already on Cloudflare's network, rather than to define what any given citation or query result should be worth. That distinction separates today's announcement from the original Pay Per Crawl launch, where Cloudflare's own dashboard let publishers set a domain-wide price per crawl directly.

Timeline

  • July 1, 2025: Cloudflare launches Pay Per Crawl in private beta, letting publishers charge AI crawlers per fetch using HTTP 402 Payment Required responses, and declares the date Content Independence Day
  • August 28-29, 2025: Cloudflare expands AI Crawl Control with customizable HTTP 402 responses and publishes data showing Anthropic's crawler accessed 38,000 pages for every one referred visit in July 2025
  • July 22, 2025: Pew Research Center publishes findings that users click AI Overview source links in just 1% of visits, and click traditional results 8% of the time when a summary appears versus 15% without one
  • September 2025: Cloudflare data shows training-related crawling reaching nearly 80% of all AI bot activity, up from 72% a year earlier
  • April 2, 2026: Cloudflare and ETH Zurich publish joint research documenting how AI crawler traffic breaks standard web caching assumptions
  • June 5, 2026: Cloudflare Radar data shows bots accounting for 57.4% of all HTML web traffic, with AI training crawlers alone at 50.6%
  • July 1, 2026: Cloudflare announces the shift from Pay Per Crawl toward Pay Per Use, launches a freshness-signal research program, and names Ceramic.ai and You.com as initial commercial partners

Summary

Who: Cloudflare, the internet infrastructure company that sits behind more than 20% of websites on its network, together with named commercial partners Ceramic.ai and its founder and chief executive Anna Patterson, and You.com.

What: Cloudflare announced a research program using freshness and traffic signals to help AI answer engines reduce unnecessary crawling, alongside a shift from its year-old Pay Per Crawl system toward what it calls Pay Per Use, under which publishers are paid when their content actually appears in an AI search result rather than each time a crawler fetches a page.

When: The announcement was published today, July 1, 2026, marking the second annual Content Independence Day that Cloudflare has used to introduce changes to its handling of AI crawler traffic, one year after the original Pay Per Crawl launch on July 1, 2025.

Where: The changes apply across Cloudflare's global network, which the company says underlies more than 20% of websites worldwide, with the two commercial experiments involving Ceramic.ai and You.com as the named initial partners.

Why: Cloudflare states that more than half of crawl traffic from bots it classifies as legitimate re-fetches pages that have not changed, a pattern it says wastes compute for AI companies and hosting costs for publishers, while separate research including a 2025 Pew Research Center study has documented steep declines in the traffic publishers receive when AI-generated summaries appear in search results.