Cloudflare unveils tools to give publishers control over AI scraping
New AI Audit feature allows website owners to analyze and manage how AI models access their content, with plans for a marketplace.
On Monday, September 23, 2024, Cloudflare announced the launch of AI Audit, a new set of tools designed to give website owners greater control over how artificial intelligence (AI) models access and use their content. This move comes as AI companies increasingly scrape websites to train large language models, often without compensating content creators.
According to Cloudflare CEO Matthew Prince, AI bots are scanning websites thousands of times daily, potentially causing "significant destruction of value for businesses large and small." The AI Audit tools aim to address this issue by providing website owners with detailed analytics on AI bot activity and the ability to block unwanted scraping.
Key features of AI Audit include:
- Free automated control of AI bots, allowing website owners to block all AI scraping with one click.
- Analytics dashboard showing why, when, and how often AI models access a website.
- Tools to help content creators better protect their rights when negotiating with AI model providers.
- Plans for a future marketplace where website owners can set prices for AI companies to access their content.
Prince stated, "Content creators and website owners of all sizes deserve to own and have control over their content. If they don't, the quality of online information will deteriorate or be locked exclusively behind paywalls."
The AI Audit tools are now available to existing Cloudflare customers through their dashboard. Website owners can also join a waitlist for upcoming features that will allow them to set prices for AI companies to access their content.
This move by Cloudflare comes amid growing concerns about the impact of AI on content creation and copyright. Earlier in 2024, AI-powered search startup Perplexity faced accusations of scraping websites that had explicitly indicated they did not want to be crawled.
Some website owners have reported that AI bot scraping has been so intense it felt like a distributed denial-of-service (DDoS) attack, potentially increasing their cloud computing costs and impacting service performance.
While some large publishers have struck licensing deals with AI companies like OpenAI, most website owners currently receive no compensation when their content is used to train AI models. Cloudflare's planned marketplace aims to democratize this process, allowing even smaller websites to potentially monetize their content's use in AI training.
The development of these tools reflects the complex and evolving landscape of AI and content creation. As AI models become increasingly sophisticated and reliant on vast amounts of data, questions of fair compensation and content rights are likely to remain at the forefront of discussions in the tech industry.
Cloudflare plans to provide more details about its AI-related initiatives during its upcoming Builder Day Live Stream event on September 26, 2024, at 11 am PT.
Key facts
Announcement date: September 23, 2024
Product name: AI Audit
Key features: Analytics on AI bot activity, one-click blocking of AI bots, tools for rights protection
Future plans: Marketplace for setting content access prices for AI companies
Availability: Now accessible for existing Cloudflare customers
Upcoming event: Builder Day Live Stream on September 26, 2024, at 11 am PT
Cloudflare CEO: Matthew Prince
Cloudflare NYSE ticker: NET