FacebookExternalHit is the name Meta Platforms attaches to the software that fetches a web page when somebody shares its address inside Facebook, Instagram, Messenger or one of the company's social plugins. The program reads the page, pulls out a title, a description and an image, and passes those items to whichever app has to draw the preview card. It does not rank pages, index them for a search engine, or feed them to a language model. Its entire job is to make a pasted URL look like something worth tapping.

The crawler exists because a raw link is a poor advertisement for itself. A string of characters ending in a numeric slug tells a reader nothing; a card carrying a headline, a sentence and a photograph tells them a great deal. Every major messaging platform runs an equivalent, but Meta's arrived first at scale and the metadata standard it reads became the standard the others read too.

Sharing triggers the fetch

The behavior that separates FacebookExternalHit from a conventional crawler is that it has no schedule. Googlebot maintains a queue of known addresses and revisits them on a cadence calculated from observed change. FacebookExternalHit waits. A request arrives because a human pasted a link into a status box or a Messenger thread, and the fetch follows within seconds.

Log volume therefore tracks sharing activity rather than site size. In a raw access log the pattern looks pathological: dozens of near-identical requests for one URL in a compressed window from many different addresses, then silence. It is close to the opposite, since each request began with a person deciding the page was worth passing on. That also makes the crawler a signal conventional analytics miss: a link shared into a private Messenger thread generates a fetch but no referrer, no session and no pageview, so server logs are the only place it appears.

What the crawler reads, and the limits on reading it

Meta's webmaster documentation sets out a short list of server-side requirements, each encoding a constraint on how previews turn out. Open Graph properties have to appear within the first megabyte of the document; anything past that point is cut off and never seen. Because the tags conventionally sit in the head element the limit rarely binds, though enormous inline scripts placed ahead of them can. The server must support gzip and deflate encodings, and the page has to be retrievable within a few seconds or Meta states the content cannot be displayed. The crawler may also send a Range header requesting a byte span, which a server should either honor precisely or ignore altogether.

The tags themselves are the Open Graph vocabulary: og:title, og:description, og:image, og:url and og:type. Supplying og:image:width and og:image:height lets Meta render the card on first share rather than downloading and measuring the file asynchronously. Where tags are absent the crawler guesses from the HTML, which is how pages end up with a download button as their preview image. One failure mode recurs often enough to name: tags written by client-side JavaScript are invisible, because the crawler parses HTML source rather than executing scripts.

Caching, and why previews go stale

Meta stores what it scrapes. Reporting from webmasters and social tooling vendors consistently puts the retention window at roughly 30 days, and re-sharing a URL neither resets it nor triggers a new fetch. Correcting an og:image on a page shared last week therefore changes nothing about what people see, because Meta is serving a copy. The Open Graph vocabulary includes og:ttl to declare how long that copy stays valid, but operators have documented cases where it is disregarded along with standard HTTP Expires headers.

The Sharing Debugger offers a way out, displaying the tags Meta actually read and carrying a control that forces a fresh scrape, with a Batch Invalidator for multiple addresses and a Graph API equivalent for deployment pipelines. HTTP and HTTPS versions of a URL are cached separately, as are the page and its image file.

Identifying the requests, and proving they are real

Two user agent strings are documented. The long form reads facebookexternalhit/1.1 followed by a parenthesized reference to facebook.com/externalhit_uatext.php; the short form is the token alone. A related string, facebookcatalog/1.0, covers product catalog fetching. The version number has sat at 1.1 for well over a decade, making it useless for telling deployment generations apart.

Verification is thinner than the alternatives. Google publishes JSON files of crawler IP ranges and refreshes them daily; OpenAI publishes prefix lists. Meta publishes neither, directing operators instead to confirm through whois or reverse DNS that a requesting address sits inside autonomous system 32934.

The gap matters because the string is trivially forged and also used legitimately by a party that is not Meta. Apple's iMessage previews send a user agent containing both facebookexternalhit/1.1 and Twitterbot/1.0, as bot detection vendors have documented. Substring matching therefore catches Meta, Apple and anyone spoofing either.

Origin and evolution

Facebook introduced the Open Graph protocol at its f8 developer conference in April 2010, and the specification spread far beyond the company that wrote it. Apple built it into iMessage from 2016 as Rich Links, and Slack, Discord and Telegram render previews from the same tags. The crawler dates to the same period, though Meta has never published a first-appearance date.

The name outlasted the company that coined it. Facebook Inc. became Meta Platforms in October 2021, yet the token kept its spelling and the reference URL still points at a facebook.com path. The family around it expanded sharply from 2024 as Meta built out AI products: Meta-ExternalAgent for training and indexing, Meta-ExternalFetcher for user-initiated retrieval, Meta-WebIndexer for Meta AI search, Meta-ExternalAds for advertising. Research by 51Degrees classifying the family by purpose in February 2026 put FacebookExternalHit alone in the preview category.

Why the preview matters commercially

Meta's value as a traffic source has fallen a long way. Chartbeat and Similarweb data covering 792 news and media sites, reported by Press Gazette in May 2024, found Facebook referrals down 50% year on year and 58% from March 2018, a collapse in referral volume PPC Land covered that month. For the 316 smallest publishers in the sample, referrals stood at 2% of their March 2018 level.

Against that backdrop the card carries more weight per share, not less. It is the only element of a Meta post a site operator controls directly, and a broken card converts an intentional share into a plain blue URL. The trade-off runs one way: blocking the crawler forecloses previews without conferring any protection against AI training, which a different token handles.

Limitations, criticisms and disputes

The sharpest open dispute concerns whether the crawler honors robots.txt. Meta's documentation says it does, with an exception for security and integrity checks such as malware scanning, and adds that rule changes can take up to 24 hours to register. Several bot-tracking services disagree bluntly: BotSights states that the crawler does not respect robots.txt, reasoning that preview bots must reach the page to render a card. Others describe compliance as unclear. The positions are not reconcilable from public evidence, and the security carve-out is broad enough that an operator cannot distinguish a permitted bypass from a plain one.

Server load is the second recurring complaint. Webmasters reported in 2024 that the crawler was hitting origins in repeated bursts, one describing spikes of roughly 400 requests per second from 20 to 30 addresses inside Meta's netblocks every 45 to 60 minutes. Because those requests come from a wide address pool, caching keyed on client IP does not absorb them.

The crawler has twice been turned into a weapon. In April 2014 researcher Chaman Thapa loaded Facebook's Notes feature with links to large files on a target server, producing more than 800 Mbps of amplified traffic from Facebook's own infrastructure. DataDome later documented Meta's preview endpoints serving as a scraping proxy. Both were fixed, but they show why traffic verifiably originating from Meta is not automatically traffic Meta intended.

Disambiguation

Meta-ExternalAgent is the training and indexing crawler introduced in July 2024. Blocking it stops content flowing to Llama and Meta AI, and has no effect on previews.

Meta-ExternalFetcher retrieves single pages when a user of a Meta AI product asks for them. Meta documents that it may bypass robots.txt because the fetch is user-initiated, a carve-out distinct from the preview crawler's security exception.

Meta-WebIndexer exists so Meta AI can cite and link to content in generated answers, which makes it a citation decision rather than a preview one.

iMessage previews are not Meta traffic at all, despite carrying the same substring. Nor is Facebot, a legacy token still listed in third-party blocking guides but absent from Meta's current documentation.

Recent developments

The context around every named crawler shifted during 2026. Cloudflare Radar data showed automated requests passing human ones for the first time, reaching 57.5% of HTML traffic by June, a figure IAB Australia cited in the bots and crawler guidance it released on July 28.

FacebookExternalHit sits outside that category, and blocking data reflects it. A monthly analysis of robots.txt files parsed across Cloudflare's network found that in June 2026 Googlebot and facebookexternalhit were the only two agents to lose named references while the sample grew, read by the analysts as site owners withdrawing generic disallows aimed at search and preview bots even as they added rules against AI training tokens.

Verification is meanwhile being rebuilt around cryptography rather than address ranges, through Web Bot Auth, Ed25519 signing keys and Cloudflare's registry format for discovering public keys at scale. A crawler verified only by a whois lookup against an autonomous system number looks dated against that.

Timeline

  • April 2010: Facebook introduces the Open Graph protocol at its f8 developer conference, defining the og: metadata vocabulary that preview crawlers read.
  • April 2014: Researcher Chaman Thapa demonstrates that Facebook Notes can amplify traffic into an attack exceeding 800 Mbps against a third-party server.
  • 2016: Apple builds Open Graph parsing into iMessage as Rich Links, sending a user agent that also contains the facebookexternalhit substring.
  • October 2021: Facebook Inc. becomes Meta Platforms; the crawler token and its reference URL are unchanged.
  • July 2024: Meta launches Meta-ExternalAgent for AI training and indexing, separating that function from the preview crawler.
  • September 2024: Webmasters publicly report repeated request bursts from the preview crawler, with figures around 400 requests per second from Meta netblocks.
  • May 2025: The IETF draft for Web Bot Auth is published, beginning the shift toward cryptographic crawler verification.
  • December 2025: Meta signs seven multi-year AI content licensing agreements with news publishers.
  • February 13, 2026: 51Degrees publishes a purpose classification of Meta's crawler family, placing FacebookExternalHit alone in the preview category.
  • June 3, 2026: Cloudflare reports automated requests at 57.5% of HTML web traffic, the first recorded machine majority.
  • June 2026: Cloudflare robots.txt sampling finds Googlebot and facebookexternalhit are the only tracked agents losing named references as the parsed corpus grows.
  • July 28, 2026: IAB Australia publishes bots and crawler guidance citing the traffic crossover and the composition of AI crawling.

Summary

Who: Meta Platforms operates the crawler. Site operators, publishers and anyone whose pages get shared into Facebook, Instagram, Messenger or WhatsApp are on the receiving end, and bot management vendors sit between the two.

What: FacebookExternalHit is a link preview crawler that fetches a page when its URL is shared inside Meta's apps, reads the Open Graph metadata, and caches a title, description and image for the preview card. It identifies itself as facebookexternalhit/1.1 and is verified through autonomous system 32934.

When: The Open Graph protocol it reads dates to April 2010, and the user agent has carried the same 1.1 version number ever since. Meta separated its AI crawling into distinct tokens starting in July 2024.

Where: Across the public web, wherever a page is reachable and its address gets pasted into a Meta surface, including private Messenger and WhatsApp threads that produce no measurable referral.

Why: A shared link without a preview card is a string of characters, and the difference in engagement is the reason the crawler exists. Its documented purpose is narrow, but its robots.txt status is contested, its verification method has not kept pace with cryptographic alternatives, and its request pattern imposes real costs on origins receiving it in bursts.