IAB Tech Lab this week released guidance on AI system bot and crawler management strategies, opening a public comment period that runs through June 25, 2026. The document addresses a gap that became apparent after the organisation published its CoMP API V1 earlier this year - many publishers have yet to formalise any strategy for dealing with non-human traffic.

The announcement, dated May 27, 2026, marks a concrete step in an industry effort to bring structure to a problem that has grown substantially more complex as AI systems multiply. According to IAB Tech Lab, the guidance outlines the range of approaches content owners can take to manage access from bots and crawlers, together with the trade-offs and operational costs involved in each. The organisation describes its target audience for the document as non-technical leadership - executives who need enough grounding in the options to ask the right internal questions, rather than engineers who would implement the controls.

What the guidance covers

The document was developed by the CoMP Working Group, the same body that produced the CoMP API V1. According to IAB Tech Lab, the motivation was straightforward: after the API specification was published, it became clear that bot and crawler management sits outside the API's technical scope. Publishers needed a companion resource to address that adjacent layer.

The guidance covers what IAB Tech Lab describes as the full range of available approaches - from open access through to graduated controls and more restrictive postures. Each strategy is presented alongside its practical advantages and trade-offs, including estimated operational complexity and resource requirements. The organisation frames the document as a tool for reducing risk, lowering operating costs, and helping content owners make decisions with greater confidence.

Bot management as a discipline has become considerably more demanding over the past two years. Non-human traffic now dominates most of the open web, and AI agents and crawlers have accelerated that shift. The challenge is no longer simply identifying bots - it is deciding which bots to allow, under what conditions, and at what cost.

According to IAB Tech Lab, the guidance is designed specifically to enable partnerships between AI systems and content owners. That framing is deliberate. A blanket blocking posture is no longer straightforward to justify: research published in early 2026 found that news publishers who blocked AI crawlers lost between 7% and 23% of weekly website trafficwithin weeks of implementing the restrictions, with declines appearing in human browsing data rather than just automated metrics.

The CoMP API connection

The bot management guidance is explicitly positioned as a support layer for CoMP API adoption. The CoMP specification, released on March 10, 2026, established a technical framework requiring AI systems to have commercial agreements in place with publishers before any content crawling occurs. The public comment period for that specification closed on April 9, 2026.

The relationship between the two documents is functional. Publishers considering CoMP adoption need to have made prior decisions about their overall bot strategy - which categories of non-human traffic to accept, which to monetise, and which to block entirely. Without that foundation, deploying a monetisation protocol becomes harder to operationalise. The new guidance is meant to help organisations work through those foundational decisions systematically.

According to Anthony Katsur, CEO of IAB Tech Lab: "Giving content owners clear, practical guidance is key if we want adoption to move forward in a meaningful way. This work helps simplify a complex area so companies can make decisions that fit their business while supporting a more sustainable marketplace."

Shailley Singh, EVP and COO of IAB Tech Lab, expanded on the structural challenge: "Content owners are being asked to make important decisions quickly, often without clear frameworks. This guidance helps break down the options so they can choose an approach that aligns with their goals."

Industry voices on the document

The release drew comment from several organisations that contributed to the guidance's development, each highlighting a different dimension of the problem.

Scott Messer, Principal and Founder of Messer Media, described the state many publishers find themselves in: "This IAB Tech Lab guidance is the essential roadmap content owners need - a practical framework to navigate the complex maze of decisions, tech, and tools available to protect their business. By empowering publishers to identify who is really there, this guide helps them distinguish between a value-driving 'ally' and a resource-draining 'extractor'. Managing AI bots is no longer just a technical background task; it is a critical business strategy for protecting what you've built and restoring a fair exchange of value for your content."

Dave Bellous, VP of Strategy at Metal Toad, addressed the infrastructure dimension: "Non-human traffic now dominates most of the open web, and AI agents and crawlers are steepening the curve. Blanket bot-blocking is no longer a useful strategy. For publishers, advertisers, and ad-tech alike, distinguishing beneficial automation from invalid traffic through graduated, standards-based controls has become a baseline requirement. The Bot and Crawler Guidance from IAB is a much-needed tool in crafting a modern web strategy."

Tom Koch, Head of Sales for the Ads and Marketing Ecosystem at TwelveLabs, focused on video specifically: "As AI agents increasingly serve as proxies for consumers, publishers and platforms need clearer frameworks for transparency and control. The crawler management framework is an important step toward giving content owners greater visibility into how their content is accessed and used. TwelveLabs is proud to contribute as this work is foundational to ensuring video, the richest form of data, is no longer overlooked in how AI engages with the web."

The scale of the problem

Context helps explain why a formal guidance document was needed at all. AI crawlers now account for 4.2% of all HTML requests across Cloudflare's network, according to Cloudflare's 2025 year-in-review data. Training-related crawling is the dominant activity, accounting for nearly 80% of AI bot traffic. The ratio of crawling to referral traffic is extreme in some cases - Anthropic's crawler pulled 38,000 pages for every single page visit it referred back to publishers in July 2025, while OpenAI maintained a ratio of 1,091 crawls per referral.

The crawl-to-referral imbalance helps explain why publisher concerns have intensified. Traditional search engine crawlers operate on a different economic logic: they crawl content in exchange for directing human readers back to source websites. AI training crawlers break that exchange almost entirely. Some platforms have gone further. Cloudflare documented cases of AI agents masquerading as human browsers to bypass defences entirely, with researchers recording crawlers using spoofed user agents.

IAB Tech Lab's own analysis, cited across multiple prior publications, estimates that AI-driven search summaries reduce publisher traffic by 20 to 60 percent on average, with niche sites experiencing losses as high as 90 percent. The organisation has put the aggregate revenue impact at $2 billion annually from AI-driven search features alone.

The economics of blocking are not simple either. The research finding that blocking leads to traffic declines - rather than simply removing bot traffic from total counts - suggests that AI referrals have become meaningful enough that cutting them off carries measurable human audience costs. Large publishers appear more exposed to this dynamic than smaller ones, according to findings published in early 2026.

Graduated controls, not binary choices

A recurring theme in the guidance, and in the industry commentary around it, is that the choice facing publishers is not binary. Blocking all non-human traffic and allowing all non-human traffic are both operationally untenable positions. The space between them - graduated, standards-based controls that distinguish between beneficial and extractive automation - is where most practical deployments will need to sit.

That framing aligns with what IAB Tech Lab has been developing at the infrastructure level. The organisation's Agentic RTB Framework, released in November 2025, established containerised auction standards designed to accommodate autonomous buying systems. The agent registry, which reached ten entries including Amazon by March 2026, addresses the identity layer - allowing platforms to verify whether an agent is legitimate before granting it access. The bot management guidance adds a decision-making framework for content owners who need to determine their posture before those technical controls come into play.

The distinction between declared and undeclared crawlers is operationally significant. Cloudflare documented Perplexity operating both a declared crawler and an undeclared one using a generic Chrome user agent, with the undeclared crawler generating 3 to 6 million daily requests in addition to the 20 to 25 million from the declared one. That pattern is precisely what graduated controls need to address - and what a strategic framework, rather than a purely technical one, helps publishers think through.

What happens next

IAB Tech Lab has set June 25, 2026 as the public comment deadline. The guidance is available at iabtechlab.com/botmanagement. The organisation says it will continue working with members on implementation support and to encourage adoption of both the guidance and the CoMP API after the comment period closes.

For marketing professionals and media buyers, the practical stakes are clear. AI bot traffic and its measurement implications have already reached analytics platforms: Microsoft Clarity added a dedicated Bot Activity dashboard in January 2026 to help website owners separate automated from human sessions. DoubleVerify reported an 86% increase in General Invalid Traffic in mid-2025, with 16% attributed to AI-powered bots including GPTBot and ClaudeBot. Those figures affect campaign measurement, viewability verification, and audience quality assessments across programmatic supply chains.

The guidance is also relevant to how publishers approach licensing discussions with AI platforms. Establishing a documented, coherent bot management strategy is a prerequisite for entering credible commercial negotiations - not merely a technical hygiene exercise. According to IAB Tech Lab, that foundation is what makes a scalable and sustainable content marketplace possible.

IAB Tech Lab was established in 2014. According to the organisation, it is a non-profit consortium whose member community spans digital publishers, ad technology firms, agencies, and brands. Its portfolio of specifications includes the OpenRTB real-time bidding protocol, the ads.txt anti-fraud specification, the Open Measurement SDK, the VAST video specification, and the Agentic Roadmap initiative.

Timeline

Summary

Who: IAB Tech Lab, a non-profit technical standards body established in 2014, along with the CoMP Working Group and contributing industry participants including Messer Media, Metal Toad, and TwelveLabs.

What: A new guidance document on AI system bot and crawler management strategies, outlining the range of approaches content owners and publishers can take to manage non-human traffic, presented alongside the trade-offs and costs of each approach. The document is open for public comment until June 25, 2026.

When: Released on May 27, 2026. The public comment period closes June 25, 2026.

Where: Published by IAB Tech Lab and available at iabtechlab.com/botmanagement. The organisation is based in New York.

Why: Following the release of the CoMP API V1, it became apparent that most content owners lack formal strategies for managing bot and crawler access - an area outside the API's technical scope but critical to its adoption. Without a structured approach to non-human traffic, publishers cannot effectively implement monetisation protocols or enter commercial negotiations with AI platforms. The guidance is intended to reduce that gap, lower operational costs, and support the development of a commercially viable content marketplace.

Share this article
The link has been copied!