Web Bot Auth is a protocol that lets an automated client attach a cryptographic signature to the HTTP requests it sends, so the website receiving them can establish which service sent them rather than taking the sender's word for it. The bot holds a private key, publishes the matching public key at a fixed location on a domain it controls, and signs every outbound request. The site fetches that key once, checks the signature, and afterwards holds a handle it can log, rate limit, allow or block.

It exists because the two identifiers the web leaned on for decades stopped working at current volumes of automated traffic. A user agent string is free text any client can copy. Address ranges are inherited from cloud providers, shared between tenants and reassigned over time, so an allowlist built on them is stale the week after it is written.

How a signed request is built

The protocol is a thin profile on top of HTTP Message Signatures, published as RFC 9421 in February 2024. A signed request carries three header fields: Signature, holding the signature value; Signature-Input, listing what was covered and under which parameters; and Signature-Agent, pointing at where the verification key can be found.

The draft sets hard requirements on those parameters. The tag must read web-bot-auth, letting a verifier discard signatures meant for other purposes. The keyid must be a base64url JWK SHA-256 thumbprint, derived from the key material rather than chosen by the operator. Both created and expires must be present, with expiry recommended no further out than 24 hours. Cloudflare's guidance is stricter, suggesting roughly a minute, since it runs no replay checks against seen nonce values.

Signers must cover at least the authority of the target, meaning the domain addressed. Method, path, full target URI and individual query parameters are optional and each narrows the signature further. Bodies are covered only if the agent sends and signs a Content-Digest field, omitted by design because most automated traffic uses GET.

Verification happens at the HTTP layer once headers arrive, in the same position as a user agent check. It can sit at the origin or, as in most deployments, at a fronting proxy that amortises key fetches across the sites behind it.

Where the keys live

Discovery runs through the Signature-Agent value. By default it resolves as an origin, and the verifier fetches a JSON Web Key Set from a reserved path, /.well-known/http-message-signatures-directory, served over HTTPS with its own media type. Two alternatives arrived in 2026: a direct JWK Set URI and an OAuth Client ID Metadata Document.

What counts as the identifier is the URL the verifier resolved, not the key. An operator can therefore rotate signing keys, publishing the new alongside the old before dropping the old, without losing continuity with sites that already recognise it. Lookups must be keyed on the pair of URL and key, or a verifier can be tricked into attributing a request to a domain whose directory it never fetched.

There is no revocation. Removing a compromised key from the directory is the only remedy, and it takes effect at each verifier when its cache next refreshes.

Origin and evolution

Cloudflare published the proposal on 15 May 2025, arguing that addresses should be abandoned as an identity signal, and filed the first Internet-Draft that month as draft-meunier-web-bot-auth-architecture. Thibault Meunier of Cloudflare was sole author; Sandor Major of Google joined later.

Deployment moved faster than the paperwork. On 1 July 2025 Cloudflare folded message signatures into its Verified Bots Program, the same day it opened pay per crawl in private beta using HTTP 402 responses. The sequencing was not incidental: charging a crawler requires knowing which crawler is asking. A separate signed agents category followed on 28 August 2025, for automation directed by end users rather than a single company.

The standards track ran alongside. An IETF group was created in a birds-of-a-feather state on 15 June 2025 and met at IETF 123; its charter was approved on 23 October 2025, under chairs David Schinazi and Rifaat Shekh-Yusef. On 30 October 2025 Cloudflare published a registry format for discovering signing agents with Amazon Bedrock AgentCore, addressing how to find keys for the next million agents rather than the first ten.

Renaming followed in June 2026, a move to Standards Track that August, and working group adoption on 1 September 2026, ending roughly ten months in which the group owned no documents while the protocol ran in production.

Who signs and who checks

Cloudflare validates signatures across its network and requires operators to register a key directory before traffic counts as verified. Google added Google-Agent to its crawler documentation on 20 March 2026 and published verification guidance on 4 May 2026, calling its implementation experimental, signing only a subset of requests, and authenticating them as https://agent.bot.goog. Google's advice is to keep address and reverse DNS checks as the default and treat signatures as supplementary.

Amazon Bedrock AgentCore signs browser traffic with a service key, customer-specific keys planned. OpenAI's ChatGPT agent signs outbound requests, a mechanism HUMAN Security built tooling around. Vercel, Shopify and Stytch have shipped support. Payments moved early: Cloudflare, Visa and Mastercard built the Trusted Agent Protocol and Agent Pay on the same foundation in October 2025, and the scheme later carried the first agent-initiated card payment in the Netherlands.

Why it matters to the marketing community

The composition of web traffic has changed underneath every measurement the industry runs. Cloudflare Radar telemetry cited by chief executive Matthew Prince in June 2026 put automated systems at 57.4% of HTTP requests for web content against 42.6% from people, with training crawlers alone at 50.6% and search crawlers at 10.7%. IAB Australia cited 57.5% in guidance published on 28 July 2026. Requests with no person attached distort session counts, inventory volumes and the denominator under every reported conversion rate.

Identity is also the precondition for the commercial arrangements built on top of crawling. Cloudflare's crawl-to-referral ratios, published in July 2026, ran from 118 to nearly 50,000 fetches per referred visitor. Charging per crawl, paying per citation and exempting an operator from a training block all depend on knowing who is at the door. So does the reverse: a publisher that wants AI assistants on its pages needs them distinguishable from scrapers, a gap visible when scanners impersonating ClaudeBot were traced to 824 addresses and when Cloudflare documented Perplexity using undeclared crawlers.

What a signature does not prove

The specification is unusually blunt about its limits. A valid signature establishes that a holder of a key published at a resolved URL signed the covered message. It says nothing about who operates the agent, whether the agent is benign, or whether the request is authorised. Those remain site policy.

Failure modes follow. A signature covering only the authority verifies against any method, path or body sent to that domain until it expires, so it can be replayed. A directory outage takes out every request naming that operator across every verifier whose cache lapses in the same window. Fetching attacker-supplied URLs invites server-side request forgery, countered with limits on response size, key count, latency and redirect depth.

The sharper disputes concern power rather than cryptography. In a use-case draft published in April 2026, Mark Nottingham recorded that access control by bot identity is contested within the group: standardising authentication could produce a web where every bot must authenticate, raising barriers to entry and creating pressure towards centralisation in some markets. The same document notes an assertion at IETF 124 that the resource requirements would disproportionately advantage well-resourced entities. Registration compounds that, since recognition in practice means admission to a programme run by the companies already deciding what reaches an origin. A rival draft from Eric Rescorla and Richard Barnes proposes anonymous bot authentication, separating rate limiting from identity.

Interoperability is unsettled too. Cloudflare's documentation, last updated on 1 July 2026, states that it implements the earlier bare-string form of Signature-Agent and fails messages using the dictionary form from later drafts. Google's documentation describes its agent sending exactly that dictionary form. The adopted draft mandates the dictionary form for signers while letting verifiers accept the legacy shape.

Not the same as

Robots Exclusion Protocol. The robots.txt file states a site's preferences and offers no enforcement. Web Bot Auth establishes who is asking, which makes a preference enforceable, but carries no preferences itself.

Verified bot programmes. Cloudflare's verified bots list and its equivalents are commercial allowlists using signatures as one input alongside address lists and reverse DNS. Membership is a vendor decision; a valid signature is not.

RFC 9421. HTTP Message Signatures is the general mechanism for signing HTTP messages. Web Bot Auth is one profile of it, fixing the tag, key identifier format and discovery path.

Trusted Agent Protocol and Agent Pay. Commerce layers using these signatures to identify a shopping agent, then adding payment credentials and instruction handling the protocol does not define.

Recent developments

Adoption on 1 September 2026 makes the protocol a document the group collectively owns, though no publication date is set and the charter milestones for April and August 2026 passed unmet. Registry and signature agent card work remains at individual-submission stage.

Enforcement moves next. Cloudflare blocks training and agent crawlers by default on advertising-carrying pages for domains onboarded from 15 September 2026. Its bot directory now lets operators edit and withdraw entries, a change made partly because operators migrating from address allowlists to signing had to file new submissions. IAB Australia's guidance sorts every crawler into one of four verdicts, a judgement that becomes tractable only once the crawler is named.

Timeline

  • February 2024: RFC 9421, HTTP Message Signatures, is published, providing the mechanism the protocol profiles.
  • 15 May 2025: Cloudflare publishes the Web Bot Auth proposal and files draft-meunier-web-bot-auth-architecture-00.
  • 15 June 2025: An IETF group is created in birds-of-a-feather state; a session follows at IETF 123.
  • 1 July 2025: Message signatures join Cloudflare's Verified Bots Program, alongside the pay per crawl beta.
  • 28 August 2025: Cloudflare introduces a signed agents classification for user-directed automation.
  • 23 October 2025: The webbotauth charter is approved and the group becomes active.
  • 24 October 2025: Visa and Mastercard build Trusted Agent Protocol and Agent Pay on the same foundation.
  • 30 October 2025: A registry format for discovering signing agents is published with Amazon Bedrock AgentCore.
  • 20 March 2026: Google adds Google-Agent to its crawler documentation with an experimental signing reference.
  • 1 April 2026: A use-case draft records the group's internal dispute over access control and centralisation.
  • 4 May 2026: Google publishes verification guidance and identifies signed agent traffic as https://agent.bot.goog.
  • June 2026: The architecture draft is renamed to draft-meunier-webbotauth-httpsig-protocol.
  • 19 July 2026: A competing draft proposes anonymous bot authentication without identity.
  • 18 August 2026: Revision 02 moves to Standards Track and mandates Signature-Agent on every signed request.
  • 1 September 2026: The working group adopts the protocol as draft-ietf-webbotauth-httpsig-protocol-00.
  • 15 September 2026: Cloudflare's default block on training and agent crawlers takes effect for new domains.

Summary

Who: Thibault Meunier of Cloudflare and Sandor Major of Google author the specification, under an IETF working group chaired by David Schinazi and Rifaat Shekh-Yusef. Cloudflare, Akamai, Amazon, HUMAN Security, Vercel and Stytch verify signatures; Google, OpenAI, Amazon Bedrock AgentCore and commerce agents produce them.

What: A profile of HTTP Message Signatures in which an automated client signs requests with an Ed25519 private key and publishes the public half as a JSON Web Key Set, letting the receiving site verify identity instead of trusting a user agent string or an address range.

When: Proposed on 15 May 2025, in production at Cloudflare from 1 July 2025, chartered as an IETF working group on 23 October 2025, and adopted as a working group document on 1 September 2026.

Where: At the HTTP layer, verified either at origin servers or, more commonly, at the content delivery networks and bot management providers sitting in front of them.

Why: Automated traffic passed human traffic on the web during 2026, and neither user agent strings nor address ranges can tell a wanted crawler from one wearing its name. Provable identity is the precondition for access rules, crawl pricing, agentic commerce and any measurement that needs machine requests separated from people.