Google Search Central today published the second episode of The Non-Canonical Cabinet, a YouTube series in which Gary Illyes and Cherry Prommawin of Google Search Relations answer odd technical questions, and this one sets out what a robots.txt file can and cannot hold - including the point that a disallow rule governs whether a crawler enters a page, not what an AI system does with content it has already collected.

In Short

Two Google employees made a short video about the small text file websites use to tell automated bots which pages they may visit. They explained that the file only understands four kinds of instructions, that anything else, including emoji in bot names or homemade rules, is quietly thrown away, and that a "no entry" rule says nothing about how an AI company uses pages it already copied. For site owners trying to keep AI bots away, the practical point is that only correctly written rules naming each bot do anything at all.

What was published

The video, titled "What can you actually put in a robots.txt?", appeared on the Google Search Central YouTube channel with the date 21/10/2026. At the time the page was captured it showed 1,885 views and 27 likes on a channel with 809,000 subscribers. It is available at youtube.com/watch?v=-jHONLQgE-s, and the pasted transcript runs to 9 minutes and 31 seconds.

According to the description, the series is "where Cherry and Gary from Google Search Relations drag weird, edge-case technical questions asked by the community out of a dusty filing cabinet and onto your screen." The episode opens with a list of the questions it intends to answer: "Can you use ⛔ emoji to keep crawlers out? What about ASCII art chicken or non English characters in your robots.txt file? And can robots.txt tell AI bots to stay out of chicken's personal diary? Is chicken bot real?" Its own summary is five words long: "TL;DR: Keep your robots.txt beautifully boring."

The first episode, on what characters can appear in domain names and URLs, was published on October 14. In it, the speaker answering the ranking question said keywords in a URL become basically irrelevant once a page has content. The format has not changed: a scripted comedy built around a filing cabinet, a chicken and a long series of poultry jokes, with technical answers placed between them. The closing line of this episode confirms the next topic: "Next episode, we are going to talk about 404."

Two caveats apply to every quotation below. The transcript is YouTube's automatic captioning, and its errors are set out in a separate section near the end of this article. And it carries no speaker labels. Where one host addresses the other by name, the speaker can be inferred; elsewhere this article attributes lines to the hosts. The captions render Prommawin's name variously as "Terri" and "Jerry". Everything said in the video is treated here as Google's presentation of how robots.txt works, not as independent findings.

What robots.txt is, according to the hosts

The technical portion starts at 0:47. The hosts called robots.txt "the gatekeeper of your website," and defined it as "just a simple, plain text file that lives at the root of a website to tell automated bots what they can and cannot do." The captions then say the rules "apply to search engine crawlers. I training crawlers and other well-behaved crawlers" - "I training" is almost certainly "AI training" misheard.

Sequence matters in their account. "Before Googlebot or any other well-behaved crawler even looks at the content on a site, it checks this file first to see which rooms it's allowed to enter and which ones strictly off limits." The hosts also tied the file to its formal basis: "this is all defined by an internet standard RFC 9309."

That standard is relatively young for a convention that dates from 1994. The Robots Exclusion Protocol was published by the IETF as RFC 9309 in September 2022. When Illyes wrote about the standard's development last year, he said the protocol had barely changed in 25 years and that no single party could change it unilaterally. The qualifier "well-behaved" in the video carries the same limitation the 1994 convention did: compliance is voluntary.

ASCII art, comments and the 97% file

The first question taken from the cabinet was whether a robots.txt file can be decorated. "A robots.txt file is not your personal art Canvas," came the reply. "It's a plain text file." The reason given was practical: crawlers "only care about things they can actually read."

Comments are the exception. "Start your lines with hash and crawlers will completely ignore them," one host said. "So while a questionable life choice, you can paste a giant Ascii chicken in there."

One anecdote put a number on how far this can go. "I once found a website whose robots.txt file was 97% script and only 3% directives." Whether "script" is what was said, or a caption rendering of another word such as "art", cannot be confirmed from the transcript. The exchange that followed reads, as captioned: "Did it break a parser. No waste bandwidth. Yeah Gary. Isn't that your website. I have many websites." Neither the site nor the measurement was identified.

The bandwidth point has a documented ceiling behind it, though the video did not cite it. Google's documentation on how it interprets robots.txt, linked in the video description, says Google enforces a 500 kibibyte limit on the file and ignores content beyond it. RFC 9309 sets 500 kibibytes as the minimum parsing limit crawlers must support. A file dominated by comments therefore costs transfer on every fetch, and if it is large enough, rules placed after the limit would not be read at all.

Four rules, and silent discards

The core of the episode came in the chapter on parsers, beginning at 3:13. "Google follows that standard. The parser reads line by line. It looks for specific rules. User agent allow disallow, and sitemap."

What happens to anything else was stated plainly. "If you mess up a line or try to get fancy with your spacing, the system will just silently discard the line it doesn't understand and move on." The other host described it as "a simple machine. It only knows what to do with a few commands."

The example given was a rule written in plain English. "If you write don't crawl secret page Google looks at don't crawl. Realizes it's not a valid rule, ignores it, and might actually crawl your secret page anyway."

That is the consequence worth noting for any site using robots.txt as a privacy measure. An invalid line produces no error, no warning in the file, and no partial effect. The page it was meant to protect is treated as if no rule existed.

One detail in the four-rule list differs from the standard itself. RFC 9309 defines user-agent, allow and disallow. Sitemap is a widely supported extension, which Illyes himself described in March 2025 as an example of a rule that gained adoption outside the formal specification. Google's documentation lists all four as supported fields and states that other fields, such as crawl-delay, are not supported. The video's "four of them" is accurate for Google's parser, not for the RFC alone.

There is a related point the episode did not raise. Blocking a page in robots.txt prevents crawling, not indexing. In 2024 John Mueller explained that pages blocked by robots.txt can still appear in Google's results when other sites link to them, because Google never fetches the page to see a noindex tag.

Emoji and non-ASCII user-agent tokens

From 4:04 the hosts took the chicken question literally. "What if there's a crawler that's using the chicken emoji as a product token. Can I write user agent chicken emoji. Does the chicken bot stop."

The answer to whether such a bot exists was "Sadly, no." The technical answer followed. "But even if there were, user agent tokens can only have letter dash. An emoji is a multibyte UTF 8 character, and I don't know any encoding that would fit that letter dash. So it just won't work at all."

The captioned "letter dash" understates the rule. Section 2.2.1 of RFC 9309 says the product token "MUST contain only uppercase and lowercase letters ("a-z" and "A-Z"), underscores ("_"), and hyphens ("-")." Underscores are permitted, and the transcript does not mention them; whether the speaker omitted them or the captions dropped a word cannot be determined.

The same restriction covers other scripts. Asked whether that means non-English characters are excluded too, the answer was: "No chicken emoji or other non-English characters in the user agent token." Any crawler whose name contains accented Latin letters, Cyrillic, Arabic, Chinese or any other non-ASCII character has a robots.txt token that differs from its marketing name, if it publishes one at all.

Google's own documentation adds a detail that explains why malformed tokens fail quietly rather than loudly. It says that in the user-agent line "all non-matching text is ignored," so that "googlebot/1.2" and "googlebot*" are both treated as "googlebot", and that the field and its value are case-insensitive.

Emoji and non-English characters in URL paths

Paths are treated differently from user-agent tokens, and this was where the video made its most specific technical claim about Google. The hosts listed "German umlaut, the Portuguese cedilla," and "the entire different set of alphabets like Arabic, Hindi, Japanese or Chinese," before stating: "It's actually the exact same concept for emojis and non-English characters."

According to the hosts, "Googlebot handles international scripts and chicken like any HTTP client does behind the scenes, those characters are multibyte UTF 8. Google's parser is flexible and will automatically translate Chinese characters, German umlauts or emojis by encoding them."

The qualification came immediately. "It's the same thing Google does when crawling for stuff, but then other or older crawlers on the web might get confused, which means to be safe across the entire global internet infrastructure, you should use the encoded version in your robots.txt rules."

The worked example in the captions is badly garbled. As transcribed, the instruction is to "actually write your f 0% 9% nine 0% 9 four gear." The description identifies the path in question as "/🐔-gear". The chicken emoji is Unicode code point U+1F414, which in UTF-8 is the four bytes F0 9F 90 94. By PPC Land's calculation, the percent-encoded rule path would therefore be /%F0%9F%90%94-gear, which matches the sequence the captions were trying to render.

The stated risk is that a crawler unable to process the raw character "might just ignore the whole disallow line and crawl everything you try to protect."

On this point the video and the written sources agree. RFC 9309 section 2.2.2 requires that octets outside the ASCII range "MUST be percent-encoded," and gives the example of a path ending in the Japanese character ツ being written as "/foo/bar/%E3%83%84". Google's documentation, last updated on August 31, 2026, says its crawlers "automatically canonicalize raw UTF-8 rule paths into percent-encoded forms." Both the file and its rules are expected to be UTF-8; Google's documentation warns that characters outside the UTF-8 range may be ignored, "potentially rendering robots.txt rules invalid." That written guidance was summarised in PPC Land's coverage of Google's crawling infrastructure documentation update in November 2025.

The practical difference, then, is between Google and everyone else. Google's parser accepts raw UTF-8 paths. The hosts did not name which other crawlers mishandle them, and the video offers no data on how common that is.

Generative AI: access, not use

The longest-anticipated question arrived at 7:13. "Can I use a robots.txt file to tell AI models, hey, you can crawl my site, but don't use my chicken diary to train your generative models."

The answer drew a line that matters for every publisher weighing AI controls. "So as defined by RFC 9309, robots.txt is an access preference protocol. It only tells about. You can crawl this page or you shouldn't crawl this other page. It doesn't dictate what a system does with the data after it has already crawled it. So disallow bluntly means do not enter the room."

The follow-up: "It has nothing to do with looking at my chicken diary or putting it in a database." To keep a particular AI crawler out, "have to target their specific user agent tokens explicitly in the file." Invented rules will not work: "You can't write a fancy custom rule like no dash I dash chicken dash chicken dash diary. The bot's parser will just throw it in the trash."

As a description of RFC 9309, that is accurate. The standard governs whether a URL may be fetched, and nothing in it expresses a purpose limitation.

It is, however, a narrower account than Google's own practice. Google operates Google-Extended, a product token that has no crawler of its own and is set through robots.txt. Disallowing it tells Google whether content its crawlers already collect may be used to train future Gemini models and for grounding. That is a use preference expressed through a file the video describes as access-only. The token does not contradict the RFC, since it is formally just another user-agent group, but it does mean that at least one major operator, Google itself, attaches meaning about data use to robots.txt groups. The video did not mention Google-Extended.

The distinction between access and use also explains why the industry has moved beyond the file. The access model only works when crawlers identify themselves and comply. TollBit's data for the first half of 2026 found that about 15% of identified AI page-fetching agents reached URLs European sites had disallowed. A HasData test in July found that of 592 sites disallowing GPTBot, 234 still served it a live page, and that Cloudflare's September 15 change removed managed AI rules from 118 sites' robots.txt files.

Cloudflare's own response has been to separate the two questions the video keeps apart. Its update in September added an option that publishes a no-training preference while still letting Google, Apple and Microsoft search crawlers index a site. And on October 16 it said it would score Verified bots on whether they honour robots.txt and slow down after HTTP 429 responses.

One further limitation sits outside the episode's scope. Some Google agents do not consult robots.txt at all. Google-Agent, added to Google's crawler documentation in March 2026, is a user-triggered fetcher that ignores robots.txt rules because a person requested the fetch. A disallow rule for Googlebot says nothing about it.

The recap

The recap, from 8:26, listed three points. "First, well-behaved crawlers, including Googlebot, use the robots.txt file to limit crawling. Second, your robots.txt file should follow the official internet standard, RFC 90 309." The third: "use percent encoding for emojis and non-English characters in robots.txt. Rule paths to keep them safe for all crawlers."

The hosts then pointed to "the two links in the description" and asked viewers to "please keep your Ascii art out of critical directives. Let's keep it boring. Beautifully boring."

The description in fact lists three links under "Learn more": an introduction to robots.txt at goo.gle/robotstxt, Google's "How Google interprets robots.txt" specification page, and the list of Google's common crawlers.

Why this matters for marketers and publishers

Most of the episode concerns syntax that few sites will ever get wrong in these specific ways. Few publishers put emoji in user-agent lines. The value of the video lies elsewhere.

The first point is the failure mode. A robots.txt mistake does not announce itself. Lines the parser does not understand are discarded without notice, so a rule written in prose, with an invented directive, or with a malformed token simply does not exist as far as the crawler is concerned. Sites that rely on robots.txt to keep staging areas, internal search pages or faceted URLs out of crawl paths depend on every line being valid.

The second is the AI distinction. Google's own speakers have now stated on its official channel that robots.txt "doesn't dictate what a system does with the data after it has already crawled it." That puts on record what publishers negotiating with AI companies have long argued: a disallow rule is a request to stay out, not a licence term. Control over use, where it exists, comes from operator-specific tokens such as Google-Extended, from separate mechanisms, or from contract.

The third is internationalisation. For sites whose URLs contain non-ASCII characters, the video confirms that Google's parser handles raw UTF-8 paths, and advises the percent-encoded form for every other crawler. Both statements are consistent with RFC 9309 and Google's written documentation.

Caption errors and chapter mismatches

The transcript pasted for this article is YouTube's automatic captioning. The following errors and ambiguities were found; quotations above keep the captioned words.

  • "Gary Gary." at 0:00 is a repeated name in the captions.
  • "Terri" (0:08) and "Jerry" (0:22) appear to be renderings of Cherry, Prommawin's first name as given in the description.
  • "I training crawlers" (1:05) almost certainly means "AI training crawlers".
  • "97% script" (2:43) may be a mis-hearing; the chapter is about ASCII art.
  • "RFC 90 309" (3:09 and 8:36) is RFC 9309, which the captions render correctly at 1:28 and 7:25.
  • "letter dash" (4:18 and 4:24) omits underscores, which RFC 9309 also allows in product tokens.
  • "There's a chicken bot. Sadly, no." (4:13) reads as a question in the original exchange: whether a chicken bot exists.
  • "Shanghai a saucepan?" (7:47) is unintelligible in the captions.
  • "Get you a chicken emoji." (6:29) is unclear; the following line refers to the path "/🐔-gear" in the description.
  • "f 0% 9% nine 0% 9 four gear" (6:30) is the captions' attempt at %F0%9F%90%94-gear.
  • "no dash I dash chicken dash chicken dash diary" (8:12) is likely a spoken hyphenated rule name, probably "no-AI-chicken-diary", with a repeated word.
  • An empty caption line sits at 3:56.

Chapter titles mostly match their content, with three boundary mismatches. The line "A robots.txt file is not your personal art Canvas" at 1:58 belongs to the ASCII art discussion but falls inside the "What is robots.txt & RFC 9309?" chapter. The "Why percent-encode non-ASCII paths in robots.txt" chapter, listed at 6:15, begins in the middle of a sentence that started in the previous chapter. And the transition "let's talk about the big one generative AI" at 6:58 sits in the percent-encoding chapter, before "Generative AI & The New Frontier" begins at 7:06. Chapter start times in the pasted transcript (for example 2:13 for ASCII art) also differ by a few seconds from the description's list (02:10), because the transcript shows the first caption line after each chapter marker.

Timeline

Summary

Who: Gary Illyes and Cherry Prommawin of Google Search Relations, presenting on the Google Search Central YouTube channel.

What: The second episode of The Non-Canonical Cabinet, which explains that robots.txt parsers read only user-agent, allow, disallow and sitemap lines and silently discard the rest, that user-agent tokens cannot contain emoji or non-ASCII characters, that non-ASCII URL paths are best written percent-encoded, and that robots.txt controls access rather than what AI systems do with crawled data.

When: Today, October 21, 2026.

Where: YouTube, at youtube.com/watch?v=-jHONLQgE-s, with links to Google's robots.txt documentation in the description.

Why: Questions about decorating robots.txt, using non-English characters and stopping AI training had accumulated from Google's event Q&A sessions. The answers matter to publishers because invalid rules fail without warning, and because a disallow rule alone does not set terms on how already-collected content is used.