robots.txt is a plain text file, published at the root of a website, that tells automated clients which parts of the site they may request. Its formal name is the Robots Exclusion Protocol. A crawler, the software search engines and artificial intelligence (AI) companies use to collect pages, is expected to read the file before fetching anything else on a host and to skip the paths it lists. It exists because servers have no lighter way to state preferences to machines. The file is a request, not a lock: the Internet Engineering Task Force (IETF) standard behind it says its rules are not a form of access authorisation.

How the file works

The file must sit at /robots.txt, in lower case, in the top-level directory of a host, as UTF-8 encoded plain text, according to RFC 9309, the IETF Request for Comments that defines the protocol. Scope is narrow. Google's documentation states that a file at https://example.com/robots.txt covers neither http://example.com, nor shop.example.com, nor a non-standard port, so every subdomain needs its own.

Inside, the file is organised into groups. Each opens with one or more user-agent lines naming a crawler by its product token, such as Googlebot or GPTBot, followed by allow and disallow rules containing URL paths. A crawler uses the group matching its token, ignoring case, and falls back to the group addressed to the asterisk wildcard; with neither, nothing is restricted. Within a path, an asterisk matches any sequence of characters and a dollar sign marks the end of a URL.

Conflicts resolve by length. The rule whose path matches the most characters wins, and where an allow and a disallow rule are equally specific, the allow rule should prevail. In Google's worked example, allow: /p beats disallow: / for the URL /page, because the allow rule is longer. A two-line group reading user-agent: GPTBot and disallow: / removes an entire host from OpenAI's training crawler.

Failure handling matters as much as syntax. Under RFC 9309, a 400-range response such as 404 Not Found means the file is unavailable and anything may be fetched. A 500-range server error means it is unreachable, so the crawler must assume everything is disallowed, though after a long period, 30 days in the specification's example, it may treat the file as absent. Crawlers should follow at least five redirects, should not rely on a cached copy for more than 24 hours, and must parse at least 500 kibibytes, the ceiling Google enforces.

Parsers may read records outside the standard, most commonly sitemap, a line pointing to an Extensible Markup Language (XML) list of URLs. Crawl-delay, which asks for a pause between requests, never entered the specification: Google ignores it, while Bing has documented support since 2009.

Publishers and retailers write the file, increasingly through a content management system or a network provider; Cloudflare said more than 3.8 million domains used its managed robots.txt service in September 2025.

From mailing list to RFC 9309

The convention predates Google. After robots swamped servers with rapid-fire requests in 1993 and 1994, Martijn Koster, then at the British software company Nexor, proposed the file in February 1994. A consensus was reached on June 30, 1994 on the robots mailing list, according to the original text, which described the arrangement as "not enforced by anybody".

Interpretations drifted for 25 years. On July 1, 2019, Google, working with Koster, submitted a draft to the IETF and open-sourced its production parser, noting that about half a billion websites relied on the protocol, according to its announcement. A day later it said it would stop honouring unofficial rules, including noindex, from September 1, 2019, after finding their use contradicted by other rules in all but 0.001% of files. RFC 9309 followed in September 2022, written by Koster and three Google employees, Gary Illyes, Henner Zeller and Lizzi Sassman. It added error handling, caching and size limits without rewriting the 1994 rules.

The file's purpose shifted in 2023. OpenAI published GPTBot on August 7 and Google added the Google-Extended token on September 28, giving publishers named targets for AI training opt-outs. By August 2024, 35.7% of the world's 1,000 largest websites blocked GPTBot, against 5% at launch. The Data Provenance Initiative estimated that in the year to April 2024 robots.txt restrictions came to cover about 5% of tokens in three major training corpora and more than 25% of tokens from the most critical domains, according to its audit of 14,000 domains.

Why marketers deal with it

For search, the file controls fetching, not listing. Google's John Mueller explained in September 2024 that a URL blocked in robots.txt can still be indexed on the strength of external links, because the crawler never sees a noindex tag on a page it cannot fetch. "The robots.txt isn't a suggestion, it's pretty much as absolute as possible," Mueller said.

Advertising crawlers follow different rules. AdsBot-Google, which checks landing pages for Google Ads, and Mediapartners-Google, which reads pages to serve AdSense and Ad Manager ads, ignore the wildcard group entirely, according to Google's list of special-case crawlers. A file disallowing everything for all agents still admits them; naming them explicitly can switch off ad quality checks that decide whether a destination is approved.

AI raised the commercial stakes. Research from Rutgers Business School and Wharton found that news publishers blocking large language model crawlers lost about 7% of weekly traffic within six weeks, measured in human browsing data. Platforms also use the file competitively: Reddit in July 2024 shut out most crawlers while Google appeared to keep access, and Amazon added six more AI crawlers to its file on August 21, 2025.

The file is acquiring legal weight. The European Union's General-Purpose AI Code of Practice, published in July 2025, commits signatories to crawlers that follow robots.txt as specified in RFC 9309, part of procedures for honouring rights reservations under EU copyright law.

Where it fails

Compliance is voluntary and, on the evidence, partial. A study by Taein Kim and colleagues of 130 self-declared bots over 40 days found bots less likely to comply with stricter directives and AI search crawlers rarely checking the file at all, according to the paper. TollBit found 15% of AI page fetchers in Europe reached disallowed URLs. Cloudflare accused Perplexity in August 2025 of switching to undeclared crawlers once its declared bot was blocked, counting 3 to 6 million undeclared requests a day alongside 20 to 25 million declared ones; Perplexity disputed the account.

Whole categories of traffic sit outside it. Google treats fetches requested by a person as exempt, and Google-Agent, added on March 20, 2026, joined that category. OpenAI removed robots.txt compliance language for ChatGPT-User on December 9, 2025.

Its vocabulary is coarse. A disallow rule stops a fetch; it cannot say "index this but do not train on it". Because Googlebot gathers pages for classic results and AI features alike, a June 2026 requirement from the UK Competition and Markets Authority (CMA) yielded a Search Console toggle rather than a crawler rule. Nor is the file retroactive: a rule written today leaves last year's collection untouched.

Implementations also diverge. RFC 9309 treats a server error as a complete disallow; Google stops crawling for 12 hours, then uses the last good copy for up to 30 days and, with no cached copy, assumes no restrictions, according to its documentation. The specification separately warns that listing paths makes them discoverable.

Extending the format is slow by design. Illyes wrote in March 2025 that changes need consensus across the ecosystem, because "no one entity can make unilateral changes to it". Cloudflare's Content Signals line, added on September 24, 2025 with search, ai-input and ai-train values, shows the cost: Mueller said in July 2026 that it had no effect on any crawler or large language model.

Not the same as

Robots meta tag and X-Robots-Tag. Page-level instructions such as noindex that govern indexing rather than fetching, and only work if robots.txt lets the crawler reach the page.

ads.txt. An Interactive Advertising Bureau (IAB) Tech Lab file modelled on robots.txt that sits in the same root position but lists who may sell a publisher's inventory. It says nothing about crawling.

llms.txt. A 2024 proposal for a Markdown summary aimed at language models. It grants or refuses nothing, and 97% of such files received no requests in May 2026, according to Ahrefs data.

Sitemap. An inclusion list of URLs a site wants discovered. robots.txt can point to one, but a sitemap cannot exclude anything.

Recent developments

Infrastructure providers now write the file for publishers. Cloudflare's Bot Preference Sync, published on August 21, 2026, generates robots.txt directives from dashboard settings. On September 15, 2026, Cloudflare added a Disallow AI Training setting that publishes no-training preferences, such as a disallow for Google-Extended, while Googlebot, Applebot and Bingbot keep crawling for search, according to its announcement. Fewer than 1% of its sites block search bots, against 17% that block training in some way, the company said.

Standardisation continues. A draft from the IETF AI Preferences working group, written by Illyes and Mozilla's Martin Thomson and at version 05 since August 19, 2026, would update RFC 9309 with a Content-Usage rule carrying values such as train-ai=n, according to the draft. A companion vocabulary draft reached version 08 on September 13.

Courts are now testing what the file means. In publishers' New York case against OpenAI and Microsoft, OpenAI argues that copies made before publishers updated their robots.txt files were impliedly licensed. In California, Reddit's claims against Anthropic returned to state court under an order that treated Anthropic's public promise to honour robots.txt as evidence of what it knew about Reddit's access rules. A convention agreed in 1994 to spare overloaded servers now sits inside disputes over licensing.

Timeline

  • 1993 to 1994: Web robots swamp servers with rapid-fire requests and wander into unsuitable areas of sites
  • February 1994: Martijn Koster, working at Nexor, proposes an exclusion file on the www-talk mailing list
  • June 30, 1994: Consensus on A Standard for Robot Exclusion is reached on the robots mailing list
  • August 2009: Bing documents support for the crawl-delay record, which Google never adopts
  • July 1, 2019: Google submits the Robots Exclusion Protocol to the IETF with Koster and open-sources its parser
  • September 1, 2019: Google stops honouring noindex and other unofficial rules in robots.txt
  • September 2022: The IETF publishes RFC 9309, authored by Koster, Illyes, Zeller and Sassman
  • August 7, 2023: OpenAI announces GPTBot, its training crawler
  • September 28, 2023: Google introduces the Google-Extended token for AI training and grounding
  • November 2023: Google replaces the Search Console robots.txt Tester with a robots.txt report
  • July 2024: Reddit restricts most crawlers through robots.txt while Google retains access
  • July 2024: The Data Provenance Initiative reports that about 5% of C4 tokens became restricted by robots.txt in a year
  • August 2024: 35.7% of the top 1,000 websites block GPTBot, up from 5% a year earlier
  • March 28, 2025: Google's Gary Illyes sets out how the protocol could evolve through ecosystem consensus
  • July 2025: The EU General-Purpose AI Code of Practice commits signatories to crawlers following RFC 9309
  • August 2025: Cloudflare accuses Perplexity of using undeclared crawlers to bypass robots.txt blocks
  • August 21, 2025: Amazon adds six more AI crawlers to its robots.txt file
  • September 24, 2025: Cloudflare launches the Content Signals Policy for more than 3.8 million managed domains
  • December 9, 2025: OpenAI removes robots.txt compliance language for ChatGPT-User
  • March 20, 2026: Google adds Google-Agent as a user-triggered fetcher that ignores robots.txt
  • March 28, 2026: A federal judge remands Reddit's claims against Anthropic to California state court
  • April 21, 2026: Rutgers and Wharton revise their study finding a 7% traffic loss for publishers blocking AI crawlers
  • June 3, 2026: The UK CMA imposes its publisher conduct requirement and Google tests an AI opt-out toggle
  • June 23, 2026: Microsoft Clarity adds robots.txt violation detection to Bot Analytics
  • July 6, 2026: John Mueller says Content Signals directives have no effect on any crawler
  • August 19, 2026: The IETF AI Preferences attachment draft reaches version 05, proposing a Content-Usage rule
  • August 21, 2026: Cloudflare publishes Bot Preference Sync
  • September 13, 2026: The IETF AI Preferences vocabulary draft reaches version 08
  • September 15, 2026: Cloudflare introduces Disallow AI Training and schedules Managed Robots.txt for deprecation
  • September 2026: OpenAI argues in the New York publishers' litigation that copies made before robots.txt updates were impliedly licensed

Summary

Who: Website owners, publishers and retailers write robots.txt, often through content management systems or networks such as Cloudflare. Search engines, AI developers, advertising platforms and archives read it. Martijn Koster devised it; Google's Gary Illyes, Henner Zeller and Lizzi Sassman co-authored the standard with him.

What: A plain text file at the root of a host that groups allow and disallow rules by crawler name, resolved by longest match. It governs whether a page is fetched, not whether it is indexed, and carries no enforcement mechanism of its own.

When: Agreed on a mailing list on June 30, 1994, submitted to the IETF on July 1, 2019 and published as RFC 9309 in September 2022. Its role widened with AI crawler tokens from August 2023, and extensions for AI usage preferences remained in draft as of September 2026.

Where: At /robots.txt on every host, protocol and port combination, with separate files required for each subdomain, and referenced in the EU's General-Purpose AI Code of Practice and in US litigation over AI training.

Why: Servers needed a cheap way to keep robots out of places they strained or should not index. That same file now carries publishers' refusals of AI training, which is why its voluntary nature, coarse vocabulary and uneven compliance have become commercial and legal questions rather than technical ones.