Meta-ExternalAgent is the identifier Meta Platforms attaches to the automated program that reads publicly accessible web pages and passes what it finds to the company's artificial intelligence systems. A crawler is software that follows links across the internet and copies the text it encounters. This one carries a declared purpose covering two jobs at once: training the foundation models behind Meta AI and Llama, and indexing content so Meta products can surface it. Site operators meet the name in server logs and in robots.txt, the plain text file at the root of a domain that tells automated visitors which paths they may fetch.

The crawler exists because Meta needs text and the open web is the largest supply of it. Naming the collector gives publishers something to point a rule at; before a token exists, refusing a crawler means guessing at IP addresses.

How the crawler identifies itself

Meta documents two acceptable forms of the user agent string, the line of text a client sends with every HTTP request to declare what it is. The full version reads meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler); the short version drops the URL. The whole Meta crawler family carries the same 1.1 version number as of August 2026, which makes version strings useless for telling deployment generations apart.

The robots.txt token is meta-externalagent. Meta's documentation for webmasters states that changes to robots.txt can take up to 24 hours to register, so a rule added on a Monday morning may not stop requests until Tuesday.

Verification works differently from most large crawlers. Meta publishes no JSON file of crawler IP ranges, directing operators instead to confirm through a whois lookup that a requesting address belongs to autonomous system 32934, Meta's own network block. The method works, but it disadvantages small publishers without log tooling against spoofed traffic wearing Meta's name.

Where it sits among Meta's other agents

Meta's webmaster documentation lists five crawlers, and the distinctions between them determine what a robots.txt rule accomplishes. Meta-ExternalAgent trains foundation models and indexes content directly. Meta-ExternalFetcherretrieves individual links when a user asks for them, work Meta ties to "evaluating and improving agentic AI capabilities". Meta-WebIndexer improves the quality of Meta AI search results. Meta-ExternalAds crawls for advertising and business products. facebookexternalhit generates the link previews that appear when a page is shared into Facebook, Instagram or Messenger.

Two carry documented exemptions. Meta states that facebookexternalhit may bypass robots.txt during security or integrity checks such as malware scanning, and that Meta-ExternalFetcher may bypass it because the fetches happen at a user's request. Meta-ExternalAgent carries no such carve out, which places it alongside OpenAI's GPTBot and Anthropic's ClaudeBot: a bulk collector expected to honour exclusion rules.

Origin and evolution

The crawler appeared in July 2024. Meta updated its developer documentation late that month, and the change went largely unnoticed until Fortune reported it on 20 August 2024. Dark Visitors, which tracks scraper activity, described the new agent as analogous to GPTBot. At that point roughly 2% of the world's most popular websites blocked Meta-ExternalAgent, against about 25% for GPTBot, a gap explained by the eleven month head start OpenAI's token had in publisher blocklists.

Meta's position at launch was that nothing had changed in substance. A company spokesperson said Meta trains its generative models "on content that is publicly available online". The context was capital expenditure: Meta had guided to spending as much as $40 billion during 2024 on AI infrastructure, and models at that scale consume text faster than licensed sources can supply it.

The family has since expanded. Meta-WebIndexer, absent from the original set, now sits alongside the training crawler, while FacebookBot, the legacy speech recognition agent still listed in many third party blocking guides, no longer appears on Meta's documentation page.

How much of the web it touches

Absolute figures for a single crawler are hard to come by, since only network operators see traffic at that scale. Cloudflare measurements cited in PPC Land's reporting on AI agents disguising themselves as human visitors put Meta-ExternalAgent at 0.9% of AI crawling activity in July 2024, rising to 7.5% by July 2025.

Cloudflare's 2025 year in review found AI bots originated 4.2% of HTML requests across its network during a year in which global internet traffic grew 19%, against 43.5% from human users. Meta-ExternalAgent was named among the highest volume AI crawlers alongside GPTBot, ClaudeBot, Bytespider, Amazonbot and Applebot. A Cloudflare breakdown by crawl purpose published on 28 August 2025 found training accounted for nearly 80% of AI bot crawling; by early June 2026 training crawlers alone represented 50.6% of traffic to HTML content.

Blocking rates lag those volumes. A March 2026 sample of 4,047 robots.txt files parsed across Cloudflare's network found Meta-ExternalAgent named in 3.82% of AI targeting disallow rules, behind GPTBot at 5.52%, CCBot at 5.08%, ClaudeBot at 4.88% and Google-Extended at 4.44%. The token never appeared in an allow rule.

Why it matters for marketers and publishers

For anyone whose commercial model depends on people arriving at a page, the crawler is one half of an exchange that has stopped balancing. Cloudflare figures from July 2026, cited when Microsoft shipped a Clarity card ranking AI operators by scrape-to-referral ratio, showed ratios running from 118 crawls per referral at the low end to nearly 50,000 at the high end.

That arithmetic has driven two commercial responses. Infrastructure providers built toll booths: Cloudflare's pay per crawllaunched on 1 July 2025 using HTTP 402 Payment Required responses and Ed25519 signed requests, then shifted a year later to paying per citation after finding that more than half of bot crawl traffic re-fetched unchanged pages. And Meta started buying what it had been collecting, signing licensing agreements in December 2025 with CNN, Fox News, the USA Today Network, Le Monde Group and others, following a 2024 arrangement with Reuters. Payment terms were not disclosed.

Blocking is not free either. Publishers that shut out AI crawlers lost roughly 7% of weekly site traffic within six weeks, a figure cited in coverage of Cloudflare's crawler transparency rules.

Limitations, criticisms and disputes

The central complaint concerns the token's dual mandate. Meta-ExternalAgent covers training and indexing under one name, so a publisher that wants to appear in Meta AI answers but not in the next Llama training run has no way to express that. Google separates Googlebot from Google-Extended. OpenAI separates GPTBot from OAI-SearchBot. Meta separates nothing.

Standards work aimed at the gap remains unfinished. The IETF AI Preferences working group's vocabulary draft reached version 07 on 19 August 2026, defining train-ai and search as distinct usage categories, but the document expires in February 2027 and states plainly that several sections lack consensus.

Compliance claims are contested. PPC Land's report on a leaked Meta scraping list, based on internal documents given to Drop Site News on 6 August 2025, described harvesting from roughly 6 million websites including about 100,000 of the most trafficked domains, with allegations that Meta's internal Web Crawler tool bypassed site protections and reached content through content delivery networks. Measurement of the category supports scepticism: TollBit found that 15% of AI page fetchers in Europe reached disallowed URLs, counting any request to a disallowed path as a bypass regardless of what operator documentation claims.

Robots.txt also cannot undo anything. A rule added in 2026 has no effect on text collected in 2024, and the copyright position on that earlier collection is unsettled. In Kadrey v. Meta Platforms, Judge Vince Chhabria granted Meta summary judgment on 25 June 2025 over 666 books belonging to 13 authors, while writing that in many circumstances copying protected works to train generative models without permission will be illegal. That ruling settled little beyond those claims.

Disambiguation

Meta-ExternalFetcher. Fetches one link at a time in response to a user prompt and may ignore robots.txt on that basis, where the training crawler runs in bulk on Meta's own schedule. Blocking one does not block the other.

facebookexternalhit. Generates share previews, thumbnails and headlines when a link is posted to a Meta app. Blocking it degrades how a publisher's links look in social feeds, a cost with no relation to AI training.

Meta-WebIndexer. Also feeds AI systems, but exists to improve Meta AI search results and citations rather than to build training corpora.

GPTBot. A parallel training crawler from a competitor. It arrived in August 2023, publishes IP ranges as downloadable files, and is separated from OpenAI's search agent. Meta's token does more jobs with less granularity.

Recent developments

The most visible change of 2026 involves the newer sibling rather than Meta-ExternalAgent itself. Crawl log analysis put Meta-WebIndexer at roughly 2.2% of tracked AI crawler requests in mid-July 2026, around 23% between 20 and 22 July, and a peak of 37.8% on 9 August 2026. That seventeenfold rise inside a month took the agent from about one request in fifty to more than one in three. Meta's crawling has not slowed; it has redistributed.

Regulatory pressure is arriving on a separate track. New York's Assembly Bill A11292, passed by the Senate on 2 June 2026 by 60 votes to one and by the Assembly three days later, would require crawlers reaching New York news sites to disclose their identity, version and operator, and to state all uses the content could be put to, with civil penalties reaching $15,000 a day. A single token covering training and indexing is an awkward fit for that. Cloudflare shipped Bot Preference Sync during the week of 24 August 2026 with four transparency requirements for mixed purpose crawlers, and will block agent and training category crawlers by default on ad-carrying pages for newly onboarded domains from 15 September 2026.

France's Autorité de la concurrence separately ordered Meta on 8 July 2026 back into neighbouring rights negotiations with French press publishers, a case about snippets rather than training that lands on the same question of what Meta owes the publishers its crawlers read.

Timeline

  • August 2023: OpenAI publishes GPTBot, establishing the model of a named, blockable training crawler.
  • July 2024: Meta launches Meta-ExternalAgent and updates its webmaster documentation late in the month.
  • 20 August 2024: Fortune reports the crawler's existence; roughly 2% of top sites block it, against about 25% for GPTBot.
  • 25 June 2025: Judge Vince Chhabria grants Meta summary judgment in Kadrey v. Meta over 666 copyrighted books.
  • 1 July 2025: Cloudflare opens pay per crawl in private beta, using HTTP 402 responses.
  • 6 August 2025: Internal documents given to Drop Site News describe harvesting from about 6 million websites.
  • 28 August 2025: Cloudflare reports training accounts for nearly 80% of AI bot crawling.
  • December 2025: Meta signs AI licensing agreements with CNN, Fox News, USA Today Network, Le Monde Group and others.
  • December 2025: Cloudflare's year in review puts AI bots at 4.2% of HTML requests, naming Meta-ExternalAgent among the largest.
  • 30 March 2026: A sample of 4,047 robots.txt files finds Meta-ExternalAgent in 3.82% of AI targeting disallow rules.
  • 2 June 2026: New York's Senate passes A11292 requiring crawler self-identification to news sites.
  • 8 July 2026: France's Autorité de la concurrence orders Meta back into neighbouring rights negotiations.
  • 9 August 2026: Meta-WebIndexer peaks at 37.8% of tracked AI crawler requests.
  • 19 August 2026: The IETF AI Preferences vocabulary draft reaches version 07, defining train-ai and search.
  • 24 August 2026: Cloudflare ships Bot Preference Sync with transparency requirements for mixed purpose crawlers.
  • 15 September 2026: Cloudflare's default blocking of agent and training crawlers on ad-carrying pages takes effect for newly onboarded domains.

Summary

Who: Meta Platforms operates the crawler. Publishers, ad-supported media owners and any site with content worth reading are on the receiving end, and infrastructure providers such as Cloudflare sit between the two.

What: Meta-ExternalAgent is a web crawler that identifies itself as meta-externalagent/1.1 and collects publicly accessible pages to train Meta's foundation AI models and to index content for its products. It responds to the robots.txt token meta-externalagent and is verified through autonomous system 32934.

When: Launched in July 2024, disclosed in Meta's developer documentation later that month and reported publicly on 20 August 2024. Its share of AI crawling grew from 0.9% to 7.5% in its first year.

Where: Across the open web, on any domain that has not excluded it, with the resulting data feeding Meta AI, Llama and the assistant surfaces inside Facebook, Instagram, WhatsApp and Messenger.

Why: Training large models requires text at a scale licensing alone cannot supply, so Meta collects it directly. The token exists so that site owners can refuse, though a single identifier covering both training and indexing gives them a blunt instrument, and refusal carries measurable traffic costs of its own.