Cloudflare published a post on August 28, 2026 opening the operator side of BotBase, the directory of known bots and agents it introduced two months ago, adding submission tracking, editing and an automated review pipeline built after the annual volume of new bot submissions rose roughly seven times since 2023.

The post, written by Julian Laxman on the Cloudflare Blog, describes a change that is narrow in surface area and consequential in effect. Until August 28, a company running a crawler could fill in a form and then wait indefinitely, with no record of what happened next. Support email was the only route to an answer. That arrangement is now replaced by a tab showing status, reasons and history.

BotBase itself is not new. It arrived on July 1, 2026, alongside a broader set of policy changes that PPC Land documented at the time, when Cloudflare replaced its binary framing of AI bots with a taxonomy built on behaviour rather than label. That release gave website owners a searchable database of tracked crawlers and, separately, a dashboard measuring how often each operator fetched a page against how many readers it returned. What it did not give was any corresponding visibility to the companies on the other end of the connection.

A form that lived in the wrong place

Location was the first problem addressed. The submission form previously sat under Manage Account and then Configurations, a path that tied a bot to a billing account without acknowledging any connection to the wider bots ecosystem. According to Cloudflare, the form has moved to Protect and Connect, then Application Security, then BotBase, placing it next to the trust and bot tooling operators already use. Access is open to all customers directly from the Cloudflare dashboard.

The new screen splits into three parts by use case. A bots directory allows browsing, searching and filtering across the catalogue Cloudflare already tracks, the same set exposed publicly through Cloudflare Radar. A submission form handles new entries. A submission history tab tracks everything an account has sent.

That third element carries the weight of the release. Cloudflare states that it spoke to many bot operators and heard consistent feedback, summarised in the post as the sense that submitting a bot feels like a black box: a form is filled, submit is pressed, and nothing observable follows.

Three statuses replace an empty queue

Submission history now assigns every entry one of three states. Waiting for review indicates the submission has been received and sits in the queue. Accepted means the bot is tracked in the directory. Rejected means something in the submission requires change, and Cloudflare says it supplies the reason together with steps that can be acted on before resubmission.

Opening an individual submission exposes its full detail. A rejected entry shows why it failed. An accepted entry that Cloudflare reclassified during review shows what was altered, which matters because classification determines how sites treat the crawler afterwards. "That's exactly the gap we're closing with this new tab," according to the post, referring to the previous need to email support merely to establish whether a submission had been looked at.

One boundary is stated plainly in the announcement and deserves attention from anyone auditing an existing entry. The record covers every bot submitted from the account starting from the launch itself. Submissions made before August 28 are not described as being backfilled into the history view. A separate filter labelled My bots, available from the bots directory screen, surfaces all bots submitted under the account currently logged in.

Editing without duplicating

Identification details drift. A crawler's operator might rehost an IP list at a new endpoint after a site redesign, or migrate from an IP allowlist to signing traffic with Web Bot Auth, the cryptographic scheme Cloudflare has been building out since May 2025 and formalised through a registry format for bot and agent authentication published on October 30, 2025 in partnership with Amazon Bedrock AgentCore.

Before this release, reflecting either change meant completing the entire form again and creating a brand-new entry, with the duplication that implies for a directory meant to be authoritative. Editing an existing submission is now possible. A submission still waiting for review can also be cancelled.

Cloudflare frames accuracy as functional rather than cosmetic. Current details are described as a key component of how a bot earns and keeps Verified status, which the company says increasingly determines whether sites across its network can allow a crawler based on its behaviour. The post also restates the limit on that mechanism: the decision on what traffic is admitted rests with each individual site owner.

Three declarations instead of one label

The intake form was rebuilt on the behaviour and content use model Cloudflare introduced on July 1, 2026. Instead of compressing a crawler into a single category, an operator now declares three separate things.

The first is behaviour. A bot might index pages for search, act as an agent on a person's behalf, collect data, train models, or support SEO tooling. Multiple behaviours can be selected rather than only the closest approximation, which addresses the mixed-purpose crawler problem directly. That problem is not hypothetical: PPC Land reported on August 21, 2026 that Cloudflare had set four disclosure requirements determining whether a mixed-purpose crawler avoids being blockedon sites that disallow training.

The second is content use, meaning what the crawler retains and reshares after access. Cloudflare distinguishes a bot skimming a page for a search snippet from one storing the same page to train a model, and asks operators to declare a level using the same Content Signals vocabulary website owners already apply to their own files. The post gives a worked example of what a site might publish in robots.txt: Content-Signal: search=yes, ai-train=no, use=reference, permitting indexing and a stored reference while refusing model training. An operator's declaration is checked against exactly that kind of preference.

The third is who runs the bot. An operator crawling the web from its own infrastructure to build its own index is classified as direct. A platform carrying traffic that other companies decided to send is an intermediary. The post illustrates the distinction with a general-purpose AI assistant fetching a page because a person typed a question into a different company's application built on that assistant's API: the assistant operator supplies the infrastructure, but the request originated in someone else's product.

Automation replaces a manual rubric

The volume argument is the sharpest number in the announcement. According to Cloudflare, the count of new bots submitted each year has grown roughly seven times since 2023, and reviewing each one by hand does not scale at that rate. Every submission previously followed a fully manual path in which a member of the team assessed it against an internal rubric and made a judgment call.

The rebuilt pipeline runs a sequence of automated checks. It tests whether the submission duplicates a bot already tracked. It tests whether the declared user-agent pattern is specific enough to identify the bot without overlapping one already registered. Most consequentially, it tests whether the claimed verification method holds: Cloudflare fetches the submitted IP list, confirms reverse DNS, or validates a Web Bot Auth signature automatically rather than assigning a person to do it.

Submissions that clear those checks can be tracked immediately. Submissions that do not are routed to the team with the specific failure already flagged, instead of arriving as an unannotated entry in a queue. Cloudflare states the practical outcome as most submissions moving faster than before, without publishing a target turnaround time or a current median.

The directory is described as holding hundreds of bots that declare behaviour and content use.

Why the timing matters for publishers and buyers

The release lands 18 days before a deadline Cloudflare set in July. From September 15, 2026, crawlers classified as Training and Agent are blocked by default on ad-carrying pages for domains newly onboarding to the network, while Search crawlers remain allowed. PPC Land covered that policy alongside the shift from charging AI companies per crawl toward paying publishers when content contributes to an answer, a change Cloudflare justified with internal data indicating that more than half of crawl traffic from bots it considers legitimate refetches pages that have not changed.

For an operator, the arithmetic is unforgiving. A crawler misclassified as Training when it also performs Search work risks blanket exclusion from a growing share of monetised pages. Under the narrowed definition, non-verified bots stay blocked by default, and the Verified label only makes a bot allowable inside its relevant category. The categories a site owner permits therefore decide access. Being able to correct a classification without submitting a fresh entry, and being told why an entry was refused, becomes materially valuable in that context rather than a convenience.

For publishers and the media buyers who trade against their inventory, the significance sits in traffic composition. Cloudflare Radar telemetry cited by chief executive Matthew Prince on June 3, 2026 put automated systems at 57.4% of HTTP requests for web content against 42.6% from people, a threshold PPC Land examined in its report that the United States originates 53.5% of global bot traffic. Requests without a person attached distort inventory counts, session metrics and the denominators underneath every reported conversion rate. A directory that operators keep current is one of the few mechanisms narrowing the gap between what a log file records and what a crawler actually does.

The counterweight is adoption outside Cloudflare's own network. Google search advocate John Mueller dismissed Content Signals on Reddit in July, saying the directives carry no effect for any crawler or large language model, a position PPC Land covered in its account of how the open web pushed back on crawler standards. Cloudflare sits behind roughly a fifth of all websites, which makes its taxonomy consequential without making it universal. Declarations collected in BotBase govern behaviour on Cloudflare-proxied properties. Elsewhere they remain assertions.

Identification has also been contested in practice. Cloudflare accused Perplexity of stealth crawling in August 2025, alleging the company modified user agents and rotated addresses to evade no-crawl preferences. Automated verification of an IP list or a signature raises the cost of that behaviour for anyone seeking directory placement, though it does nothing about traffic that never applies.

What Cloudflare says comes next

The post sets out three sequential goals. Visibility is the stated target of this launch, covering the ability to see, understand and edit submissions. Ownership and observability is described as being targeted soon, covering claims of bot ownership, management of a live directory entry, and better understanding of how websites treat a given bot. Conversation is framed as a longer-term goal, described as a route for operators to ask websites for access on the basis of demonstrated value.

No dates accompany either of the two later stages. Cloudflare closes by describing BotBase as moving from a directory built for website owners toward a place where operators participate, with the line "We are building the operator side alongside the operators who use it."

Timeline

  • July 3, 2024: Cloudflare publishes analysis showing AI bots accessed roughly 39% of the top one million internet properties on its network in June 2024, with only 2.98% blocking or challenging those requests (source)
  • May 2025: Cloudflare shares its Web Bot Auth proposal, introducing cryptographic authentication for automated traffic using HTTP Message Signatures (source)
  • July 1, 2025: Pay Per Crawl opens in private beta and Cloudflare declares the first Content Independence Day (source)
  • August 4, 2025: Cloudflare accuses Perplexity of using undeclared crawlers to evade no-crawl directives (source)
  • August 29, 2025: Cloudflare expands the HTTP 402 payment protocol with customisable responses inside AI Crawl Control (source)
  • October 24, 2025: Visa and Mastercard build Trusted Agent Protocol and Agent Pay on Web Bot Auth foundations (source)
  • October 30, 2025: Cloudflare publishes a registry format for bot and agent authentication with Amazon Bedrock AgentCore (source)
  • April 2, 2026: Cloudflare and ETH Zurich publish joint research on AI crawler traffic breaking web caching assumptions (source)
  • June 3, 2026: Radar telemetry places automated systems at 57.4% of HTTP requests for web content against 42.6% from people (source)
  • July 1, 2026: Second Content Independence Day brings the behaviour-based taxonomy, the Content Signals use field, BotBase for website owners and the Attribution Business Insights dashboard (source)
  • July 1, 2026: Cloudflare shifts from per-crawl charging toward payments tied to whether content appears in a generated answer (source)
  • July 6, 2026: Google's John Mueller states that Content Signals directives have no effect for any crawler or language model (source)
  • July 12, 2026: Cloudflare opens network signals covering roughly 20% of the web to OpenAI under a pilot (source)
  • August 21, 2026: Bot Preference Sync launches, generating robots.txt from dashboard settings and setting four disclosure requirements for mixed-purpose crawlers (source)
  • August 28, 2026: BotBase for Operators launches with submission history, status reasons, entry editing, cancellation and automated verification checks
  • September 15, 2026: Training and Agent crawlers become blocked by default on ad-carrying pages for domains newly onboarding to Cloudflare (source)

Summary

Who: Cloudflare, in a blog post written by Julian Laxman, addressing bot operators, and by extension the website owners, publishers and advertising businesses that decide which automated traffic reaches their pages.

What: BotBase for Operators, a relocated and expanded interface covering a bots directory, a rebuilt submission form declaring behaviour, content use and operator type, and a submission history tab showing Waiting for review, Accepted or Rejected status with stated reasons, alongside entry editing, cancellation of pending submissions, a My bots filter, and an automated review pipeline checking for duplicates, user-agent specificity and verification method validity.

When: Published August 28, 2026, with availability stated as immediate for all customers, two months after BotBase launched for website owners on July 1, 2026 and 18 days before the September 15, 2026 default-blocking date.

Where: Inside the Cloudflare dashboard under Protect and Connect, then Application Security, then BotBase, with the same catalogue browsable publicly through Cloudflare Radar, across a network sitting behind roughly a fifth of all websites.

Why: Annual bot submission volume has grown roughly seven times since 2023, making fully manual review unworkable, while operators reported no way to check status, understand rejection or update details without filing a duplicate entry. Accurate declarations feed Verified status, which under the narrowed July 2026 definition determines whether a crawler is allowable within a category that a site owner has chosen to permit.