Cloudflare on October 2, 2026 released a Web Search API inside AI Gateway, sending queries from AI agents to Ceramic.ai, Exa or Linkup and obliging each provider's crawler to meet the company's verified-bot rules, according to a post by Michelle Chen, Sam Else and Gabriel Massadas on the Cloudflare blog.

In Short

Cloudflare added a way for AI programs to search the live web instead of guessing page addresses and landing on error pages. It matters to developers building AI tools and to website owners, because the three search companies behind it have promised to identify their crawlers, follow site rules and show where each result came from. What changes is that a search can now be billed, logged and access-controlled through Cloudflare's existing AI setup, at the search companies' list prices with no markup added.

What Cloudflare put out on October 2

The post sits under the Birthday Week tag, the label Cloudflare gives its annual run of announcements, and arrives days after the BEACON speed dataset published on September 28, 2026. Its opening premise is a small piece of agent behaviour. According to the post, an agent that needs a live page usually guesses the URL and then makes a tool call to curl it, which explains why some web fetches come back as 404 Not Found. The authors propose what people do instead: begin with a search query.

Cloudflare describes the partnership with search providers as a way to bring "grounded intelligence via AI Gateway", with Ceramic.ai, Exa and Linkup as the first three. The technique goes by the name of grounding, and it normally forms the retrieval half of retrieval-augmented generation: documents are fetched at query time and the model writes its answer from them.

The post makes the familiar case for it. Models are trained and then frozen at a point in time, so recent events, changing APIs and fast-moving news sit outside what they know. Adding search to the inference pipeline, according to Cloudflare, places fresh, structured snippets from the web straight into the model's context. The illustration comes from Cloudflare's own catalogue: an agent building with its developer tools would miss every product released during Birthday Week unless it could look them up.

How the integration works

AI Gateway as the control layer

Web Search API is built on AI Gateway, which Cloudflare calls the flagship integration point for the product and describes as a control plane with observability, unified billing, security and access controls built in. The gateway sits between an organization's applications and the AI providers it relies on, and it gained dollar-denominated spend limits in public beta on June 5, 2026.

For search, requests appear in the normal AI Gateway logs, queries draw down the AI Gateway credit balance, and administrators decide which providers a gateway may reach. Cloudflare says it will identify partners that support Zero Data Retention (ZDR), so that the data is known not to be retained, and that it offers search at the providers' list API prices without additional markup. Bring-your-own-key (BYOK) is supported, as it is for model inference providers.

The documentation fills in the mechanics. A key is stored on the gateway under an alias. A request that names an alias that does not exist fails with a 400 error instead of falling back to credits, while a request that names none uses a key stored under the alias default if one exists and credits otherwise. Stored keys are encrypted with Secrets Store, and the key itself is never sent in the request.

Two ways to call it

The direct route is a POST to https://api.cloudflare.com/client/v4/accounts/{account_id}/ai/websearch/. The documentation specifies a Cloudflare API token with read permission for both Workers AI and AI Gateway, although the post describes the credential as an AI Gateway authentication token. The JSON body takes a query of 1 to 1,024 characters, a provider (ceramic, exa or linkup, with ceramic applied when the field is left out), a limit of 1 to 10 results (10 by default), an optional byokAlias, and options.gateway.id to name the gateway. Every account holds a gateway called default, and the account needs either loaded credits or a stored provider key.

Developers working on Cloudflare Workers can skip HTTP. A standalone binding exposes env.AI.websearch(), which takes a gateway ID, a query, a provider and a limit and returns a standard Response whose json() method yields the results. Both of the post's examples put the same question about autumn activities in Salt Lake City, one to ceramic and one to exa.

Each result carries a URL, a title and, when the provider supplies them, a description, image, favicon and last-modified date, with a metadata block recording the query, a request ID and a latency figure. Because every provider's output is normalized to that shape, switching providers means changing a single parameter.

Server tools, promised without a date

Cloudflare says native Server Tools are being built into AI Gateway, with web search among the first, so that developers no longer define the tool themselves. The post gives no date beyond saying the change is coming soon. In the interim it supplies a manual pattern. The code defines a web_search function tool, runs the model @cf/google/gemma-4-26b-a4b-it through the gateway and, if the model's first tool call is web_search, passes the model-written query to env.AI.websearch() with a limit of 5. The results are serialized into a tool message and the model runs a second time to compose the answer. No provider is set in that call, so the default applies.

One flaw is visible in the sample. The post's snippet uses model and prompt variables it never declares, whereas the documentation's version defines both, which makes the post's code an outline rather than something that runs as printed.

The three providers and their list prices

Cloudflare's documentation, last updated October 2, 2026, sets out what each provider supplies and charges.

ProviderIndex and search modeZero Data RetentionList price per 1,000 requests
Ceramic.ai (default)Own index of more than 40 billion pages; descriptions up to 8,000 charactersYes$0.25
ExaOwn index combining keyword and embeddings-based search; auto search type, page highlights returned as the descriptionNo$7.00
Linkupfast search depth, raw results with no generated answerYes$5.00

The spread between the cheapest and the dearest entry is 28-fold, but the products are not like for like. Exa returns query-relevant excerpts, Ceramic.ai returns long page descriptions and Linkup returns sourced snippets without composing an answer. For scale, an agent task that runs 20 searches would cost between half a US cent and 14 cents at list prices, arithmetic that excludes model tokens and any provider-side volume terms. With BYOK the provider bills the customer directly under its own agreement, and Cloudflare's credits drop out of the picture.

Because the default is Ceramic.ai, a client that omits the provider field lands on the cheapest entry and on one of the two that support ZDR. Ceramic.ai is also a counterparty elsewhere in Cloudflare's publisher programme: on July 1, 2026, Cloudflare said Ceramic.ai and You.com would pay publishers per query result rather than per page fetch, with Ceramic.ai's founder and chief executive, Anna Patterson, quoted in the announcement. The Web Search API post says nothing about publisher payments.

The crawler conditions attached to each provider

Cloudflare attaches conditions to inclusion. The post sets out a principle that crawlers be honest, transparent and respectful of bot rules and preferences, and that site owners hold meaningful transparency and control over how their content is used. Its search partners, according to the post, have committed to meeting Cloudflare's bot crawling standards. Two requirements follow, repeated in the documentation:

  • Verified bot compliance - the crawler a provider uses must meet Cloudflare's published requirements for Verified bots, which the documentation says includes identifying the crawler and respecting robots.txt.
  • Source attribution - every result must include a link to the location of the crawled content.

The post calls the rules net-positive for a fair Internet and says they let creators decide what to do with their data.

The bar is not new. Cloudflare folded Web Bot Auth message signatures into its Verified Bots Program on July 1, 2025, the day it opened pay per crawl in private beta, and it had rolled out Robotcop on December 10, 2024 to turn robots.txt rules into firewall rules at the network edge. What has shifted is the weight of the label. Under a narrowed definition from July 2026, non-verified bots stay blocked by default and the Verified label only makes a bot allowable inside its own category, while the number of new bots submitted each year has grown roughly sevenfold since 2023.

Identification has been the sore point. Cloudflare de-listed Perplexity as a verified bot in 2025 after documenting 3 million to 6 million daily requests from an undeclared crawler, on top of 20 million to 25 million from the declared one. Controls have kept accumulating since. Bot Preference Sync, published on August 21, 2026, generates a site's robots.txt from dashboard settings. On September 15, 2026, Cloudflare added a Disallow AI Training option and began blocking training and agent crawlers by default on ad-carrying pages of newly onboarded domains, with search crawlers still allowed.

The traffic that these rules govern is large. Cloudflare data put bots at 57.4% of HTML requests across its network by early June 2026, with training crawlers at 50.6% and search crawlers at 10.7%.

Where the product sits among grounding suppliers

Cloudflare is not building an index for this product. It is placing a billing, logging and policy layer over three third-party indexes, which sets it apart from the larger suppliers of grounding data.

Microsoft retired its Bing Search APIs on August 11, 2025 and pointed developers to a grounding product priced between 40% and 483% higher, then opened Web IQ on June 2, 2026, a suite of grounding APIs on Bing's index for which the company claims 95th-percentile latency under 165 milliseconds. Brave prices its Search API at $5 per 1,000 requests and publishes answer-quality benchmarks of its own, a vendor-supplied comparison. Exa, one of Cloudflare's three, already supplies retrieval for Firefox in Smart Window on desktop and Quick Answers on iOS.

Cloudflare's separate AI Index, opened in private beta on September 26, 2025, offers per-site search indexes with a search API under site-owner control. The Web Search API post does not mention it. The two products approach retrieval from opposite ends, one through providers' own crawls and one through indexes that site owners opt into.

Why the marketing community has a stake

The first consequence is structural. Answers from AI agents built on this endpoint will draw on three distinct indexes with different retrieval methods, and any visibility work that assumes a single index behind every answer is working from a simplified picture. Which index an agent reads depends on a parameter that, by default, resolves to Ceramic.ai.

The second concerns access. Cloudflare's categories - Search, Agent and Training - decide what its network admits on ad-carrying pages, and the post does not say which category the providers' crawlers fall into. That classification determines whether those crawlers are admitted by default on newly onboarded domains. Which one applies? The post leaves it unanswered.

The third concerns referrals. Cloudflare's attribution dashboard, opened on July 1, 2026, showed crawl-to-referral ratios running from 118 to nearly 50,000 fetches per visit depending on the operator. A rule that every result must link to its source guarantees a link in the data returned to the developer. Whether that link reaches an end user, and whether anyone clicks it, depends on how each application presents results, a matter the post does not address.

The fourth is measurement. The post describes logs and analytics on the developer side of the gateway and no equivalent reporting for site owners, a gap that stands out beside Microsoft, which pairs Web IQ with Clarity citation reporting that shows the queries AI systems used to find and cite content.

What the post leaves open

The post does not say whether the product is in beta or generally available, which plans can use it or where. It gives no latency, freshness or quality figures for any of the three providers, and the documentation's only stated limits are the 1,024-character query and the 10-result cap. It describes the crawler standards as commitments without explaining how Cloudflare checks continuing compliance, what follows a lapse, or whether each provider's crawler is already on the Verified list. And server tools carry no date.

Timeline

Summary

  • Who: Cloudflare, in a post by Michelle Chen, Sam Else and Gabriel Massadas; search providers Ceramic.ai, Exa and Linkup; developers building AI agents; and the site owners and publishers whose pages those providers crawl.
  • What: A Web Search API delivered through AI Gateway, callable by REST endpoint or Workers binding, billed to AI Gateway credits or a stored provider key at list prices of $0.25, $5.00 and $7.00 per 1,000 requests, with providers bound to Verified bot compliance and source attribution, and native server tools promised.
  • When: October 2, 2026, during Cloudflare's Birthday Week.
  • Where: On Cloudflare's developer platform, through the /ai/websearch/ REST endpoint and the env.AI.websearch()Workers binding, routed via a customer's AI Gateway.
  • Why: Agents that guess URLs hit 404 errors and rely on training data frozen at a cutoff date, so Cloudflare pairs live search with its crawler rules to give the search a control layer and site owners a defined standard.