Video crawling is the process by which a search engine finds video on the web, fetches the page that hosts it, retrieves the thumbnail and often the media file itself, and decides whether the video can appear in video results. It exists because a video is opaque to an ordinary text crawler. The bytes of an MP4 file say nothing a ranking system can read cheaply, so search engines depend on a chain of signals around the file: the page that embeds it, the markup that describes it, and a still image that stands in for it on a results page.

For Google, that work is split across several clients of the same fetching infrastructure. Googlebot fetches the page. A separate crawler, Googlebot-Video, identifies itself as Googlebot-Video/1.0 and, according to Google's crawler documentation, serves "video-related Google Search features and other products that depend on videos".

How a video gets found

Discovery starts with HTML. According to Google's video best practices documentation, last updated on December 18, 2025, Google can detect videos referenced by four elements: <video>, <embed>, <iframe> and <object>. Videos injected by JavaScript count only if they appear in the rendered HTML, which can be checked with the URL Inspection tool. Players that load only after a click, swipe or typed input are a problem, as are videos loaded through URL fragment identifiers.

Three supplementary sources help. Video sitemaps list each watch page with its video details. Schema.org VideoObject structured data describes the same video inside the page. Open Graph tags, written for social sharing, are read as a third source.

Google's documentation distinguishes three addresses that are easy to confuse:

  • The watch page, the HTML document that embeds the video, which goes in the <loc> field of a video sitemap.
  • The player URL, given as VideoObject.embedUrl or <video:player_loc>.
  • The content URL, pointing to the media bytes themselves, given as VideoObject.contentUrl or <video:content_loc>.

A video sitemap entry requires a thumbnail location, a title, a description of up to 2,048 characters and either a content or player location; Google recommends the content location where one exists. Optional tags cover duration (1 to 28,800 seconds, or eight hours), expiration date, publication date, regional restrictions and up to 32 tags. Media RSS (mRSS) feeds are accepted as an alternative, although Google says it prefers sitemaps.

What happens after the fetch

Fetching the page and indexing the video are separate events. Google says its Web Rendering Service, which executes JavaScript to build what a visitor would see, does not request images or videos, a point documented in its March 31, 2026 crawler posts. Media goes elsewhere. Gary Illyes of Google told Search Central Live in Barcelona on October 2, 2026, that images, structured data and video are pulled out and sent to a separate media indexing engine, where video indexing typically takes hours to days and, in the slowest cases, weeks to months. The figures come from an attendee's recap.

Byte limits for media are not published. Googlebot reads the first 2MB of an HTML file and 64MB of a PDF, but those caps apply to Search's text pipeline. Martin Splitt of Google said on a podcast episode dated October 13, 2026, that videos and images "go to slightly different pipelines" and that he assumed they had a larger limit, without giving a figure.

Once the media system has the video, five conditions decide eligibility, according to Google: the watch page must be indexed and "performing well in Search"; the video must be embedded on that page; it must not be hidden behind other elements; paywalled video needs paywall structured data; and there must be a valid thumbnail at a stable URL. Thumbnails must be at least 60 by 30 pixels, and at least 80% of their pixels must have an alpha value above 250, which rules out mostly transparent images. Supported formats include MP4, WebM, MOV and MKV; data URLs are not.

Stability matters more than many sites expect. Content delivery network (CDN) URLs that expire quickly can stop a thumbnail being indexed. Video previews and key moments need Google to fetch the media file, including streaming manifests such as M3U8, so blocking those files in robots.txt removes the features. Google's own guidance states that it "only indexes images and videos that Googlebot is allowed to crawl", wording repeated in an October 21, 2026 page listing seven ways to keep content out of Search.

The watch page rule

The concept that now governs most outcomes is the watch page. Google defines it as a page whose "main purpose is to show users a single video", such as a video landing page or a TV episode player. A blog post with an embedded clip, or a product page with a demonstration video, does not qualify.

On April 13, 2023, Google said video thumbnails would appear next to results only when the video was the main content of the page, according to Search Engine Land. A Google spokesperson said the judgement used "several factors, including the location of a video on the page". From December 4, 2023, the same requirement was extended to Video mode, the video tab of Google Search, and Search Console gained a reason labelled "Video is not the main content of the page", according to Search Engine Land and a Search Engine Roundtable recap.

Origin and evolution

Google announced video sitemaps in a Webmaster Central blog post by John Fisher-Ogden and Amy Wu in December 2007, according to Pulse 2.0, which reported it on December 18, 2007. Bing accepted both video sitemaps and mRSS feeds through Bing Webmaster Tools, according to a Bing Webmaster blog post by Joseph Laria.

Reporting followed much later. Google launched a Search Console report for video structured data on October 7, 2019, and a dedicated Video indexing report in July 2022. Its failure reasons include "Google could not determine the prominent video on the page" and "Cannot determine video position and size", according to Yoast. On August 6, 2022, Google stopped reading several video sitemap tags; the documentation, last updated May 20, 2026, lists <video:category>, <video:gallery_loc>, <video:price>, <video:tvshow> and two player_loc attributes as deprecated.

Timestamps became a crawling problem too. Key moments let Google link to points inside a video. Clip markup gives exact start and end times; SeekToAction tells Google the URL pattern for jumping to a timestamp and leaves the segmenting to Google. SeekToAction left beta for all sites in July 2021, according to Search Engine Journal. In May 2024, Google added GoogleOther-Video and GoogleOther-Image for research and development fetches.

Why it matters for marketers

Video crawling decides whether a publisher's own video earns traffic or the hosting platform does. Google's documentation notes that a video hosted on YouTube, Vimeo or Facebook may be indexed both on the site that embeds it and on the platform's page, and the two compete. For advertisers, the crawled video library has a second use: Google Ads documentation says "up to 10 crawled videos can be automatically added to an asset group" in Performance Max, and sourced videos became enabled by default for new campaigns in October 2026.

Crawling has also reached social video. Google began indexing public Instagram videos and Reels from professional accounts from July 10, 2025, and on July 7, 2026 it added platform properties to Search Console for Instagram, TikTok, X and YouTube.

Limitations and disputes

The watch page rule is the main complaint. John Shehata described the requirement that a page already perform well before its video is considered as the hardest part, and PPC Land called it a catch-22 for independent publishers. Richard Hearne suggested the guidelines were an indirect way of saying "YouTube, Facebook, Vimeo, or nothing". Google's documentation is explicit that an indexed watch page does not guarantee an indexed video, and it publishes no threshold for "performing well".

The YouTube question has reached court without resolution. Rumble alleged that Google ranked YouTube videos first even for Rumble's own content, but Judge Haywood Gilliam Jr. dismissed the $2 billion claim on May 21, 2025 as time-barred, without ruling on the ranking allegations.

Cost is a quieter issue. Google's documentation tells sites that host and streaming servers need "enough capacity" for crawling, and offers no published fetch size or frequency for media. For large libraries the cost of being crawled falls on the publisher, and the crawl budget logic that governs pages does not clearly describe media.

Video indexing is the step after crawling: storing a fetched video so it can appear in results. A video can be crawled repeatedly and never indexed, which is what the Video indexing report measures.

Video SEO is the broader practice of optimising titles, thumbnails, markup and pages for video results. Crawling is one technical input to it.

Video sitemaps are a discovery aid, not a crawler. They list where videos are; they do not grant access or override robots.txt.

Sourced or crawled videos in Google Ads reuse video found on an advertiser's site or channels as ad assets. The term shares vocabulary with search crawling but serves campaign automation, not organic results.

Recent developments

Most of the recent movement has come from Google talking about media pipelines rather than changing them. Illyes's Barcelona figures on October 2 and Splitt's podcast remarks on October 13 confirmed that video sits outside the HTML pipeline but left its limits undocumented. Google's common crawlers page, which lists Googlebot-Video and GoogleOther-Video, was last updated on October 15, 2026. In the same month, Performance Max sourced videos moved to opt-out, extending the commercial use of crawled video.

Timeline

  • December 2007: Google introduces video sitemaps, built by the Google Video and Google Sitemaps teams
  • October 10, 2013: Bing details video sitemap and mRSS submission in Bing Webmaster Tools
  • October 7, 2019: Search Console report for video structured data launches
  • July 2021: SeekToAction key moments markup leaves beta for all sites
  • July 2022: Search Console Video indexing report launches
  • August 6, 2022: Google stops reading several video sitemap tags
  • April 13, 2023: Video thumbnails limited to pages where video is the main content
  • April 2023: GoogleOther crawler introduced
  • December 4, 2023: Main content requirement extended to Video mode
  • May 2024: GoogleOther-Video and GoogleOther-Image announced
  • August 2024: Google revises video SEO documentation
  • May 21, 2025: Rumble's $2 billion antitrust claim against Google dismissed as time-barred
  • July 10, 2025: Google begins indexing public Instagram videos and Reels from professional accounts
  • December 18, 2025: Google video best practices page last revised
  • March 31, 2026: Google documents that its renderer does not request images or videos
  • May 20, 2026: Video sitemap documentation last revised
  • July 7, 2026: Search Console platform properties for Instagram, TikTok, X and YouTube announced
  • October 2, 2026: Illyes describes the separate media indexing engine and its timings
  • October 15, 2026: Google common crawlers page, listing Googlebot-Video, last updated
  • October 21, 2026: Google publishes methods for keeping content out of Search; Performance Max sourced videos reported as opt-out

Summary

Who: Search engines operate video crawlers, chiefly Google through Googlebot, Googlebot-Video and GoogleOther-Video, and Microsoft through Bing's sitemap and mRSS intake. Publishers, broadcasters, retailers and video platforms supply the pages, markup and files, and Google Ads reuses crawled video for advertisers.

What: The discovery, fetching and evaluation of video embedded in web pages, using HTML video elements, video sitemaps, VideoObject structured data and Open Graph tags, followed by thumbnail and media file retrieval and a separate media indexing step that decides eligibility for video results.

When: Video sitemaps date to December 2007. The watch page and main content rules took effect on April 13 and December 4, 2023, and Google described its separate media indexing timings on October 2, 2026.

Where: On publisher watch pages and CDNs that serve thumbnails and media, in Google Search results, Video mode and Discover, and in Search Console's Video indexing report.

Why: Video files cannot be read like text, so search engines rely on surrounding signals to find and classify them. The rules that result decide whether video traffic reaches the publisher's own pages or the platforms hosting the files, which is why independent publishers dispute them.