YandexBot is the automated program Yandex uses to fetch web pages so they can be added to the index behind its search engine. It exists for the same reason any search crawler does: a page that has never been downloaded cannot be ranked. What sets it apart is geography and scale. Yandex holds roughly seven in ten searches in Russia, so for any business selling into Russian-speaking markets, YandexBot decides whether a site is visible in the dominant local engine.

How the crawler identifies itself

Every request from the main crawler carries a fixed user agent string: Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots). The "Mozilla/5.0 (compatible ...)" prefix is a convention shared with bingbot and Googlebot, kept so that servers built for browsers respond normally. The version number has stood at 3.0 for years, and the link points to Yandex's public list of robots.

A second variant, YandexBot/3.0; MirrorDetector, carries the same name but does a different job. It looks for alternate addresses of a site, such as http and https versions or www and non-www hosts, so Yandex can group duplicates under one main address.

Because a user agent string can be copied by anyone, Yandex publishes a verification method. According to the Yandex Webmaster help centre, the IP address from server logs is run through a reverse DNS lookup; "all Yandex robots have names ending in yandex.ru, yandex.net or yandex.com." A forward lookup on that host name must then return the original IP address. If either step fails, the visitor is an impostor. Yandex also offers an IP check tool inside Yandex Webmaster, its equivalent of Google Search Console.

From robots.txt to the index

The crawl proceeds in a familiar order. YandexBot requests the robots.txt file at the root of a host, parses the rules that apply to it, then fetches permitted URLs discovered through links, sitemaps and submissions. Pages that need JavaScript are handled with help from a separate robot, YandexRenderResourcesBot, which loads scripts and style sheets so the page can be rendered. Yandex documents a conditional rule for it: resources are fetched even when robots.txt disallows them, provided the HTML page itself is open to the main crawler.

Two tokens matter in robots.txt. A group addressed to "Yandex" applies to all Yandex robots; a group addressed to "YandexBot" applies only to the main indexing crawler. Yandex warns that "some Yandex robots may ignore directives in robots.txt, including those for User-agent: Yandex." Its guidance also notes that once a Yandex-specific group exists, the crawler disregards the general "User-agent: *" group entirely, so rules must be repeated rather than inherited.

Crawl speed is not set in robots.txt. "From February 22, 2018, Yandex doesn't take into account the Crawl-delay directive," according to Yandex documentation. Site owners instead set a crawl rate inside Yandex Webmaster. That change predates the September 2022 publication of RFC 9309, the standard that formalised the robots exclusion protocol without ever including Crawl-delay.

A dialect of its own

Yandex supports Allow, Disallow, Sitemap and the * and $ wildcards, but it also kept a directive no other major engine adopted. Clean-param tells the crawler which URL parameters do not change page content - session identifiers, sort orders, tracking tags - so that thousands of variants are treated as one page. The syntax is Clean-param: p0[&p1&p2&..&pn] [path]. A line such as "Clean-param: sid&sort /forum/*.php" tells YandexBot to ignore the sid and sort parameters on forum scripts. Rules may run to 500 characters, matching is case sensitive, and the directive is "intersectional", meaning it can sit anywhere in the file.

The practical effect is crawl economy. A retail site with faceted navigation can produce millions of parameterised URLs; Clean-param lets Yandex collapse them before fetching, rather than discovering duplication after the fact.

The older Host directive, which named a site's preferred mirror, no longer works. Third-party robots.txt tooling, including the Nuxt robots module documentation, dates its retirement to 2018. Yandex's own help pages now point to 301 redirects and canonical tags to signal the "main site address".

Forty robots, one family

YandexBot is the best known of more than 40 user agent strings Yandex lists. Some follow general robots.txt rules: YandexImages, YandexVideo, YandexMedia and YandexMarket among them. Many do not. YandexDirect, which categorises partner sites for ad matching, and YaDirectFetcher, which checks the availability of ad landing pages, are both listed as ignoring general rules. So do the YandexMetrika robots, which serve the company's analytics product, and YandexAccessibilityBot, which may make up to three requests a second regardless of the crawl rate set in Webmaster.

Two newer entries reflect the shift to generative answers. YandexAdditional and YandexAdditionalBot process robots.txt to keep content out of "Search with Yandex AI" responses and, according to Yandex, make no indexing requests. A Disallow rule addressed to them removes a page from AI answers within roughly two to 14 days, as pages are recrawled. YandexComBot, using a separate ya.cc address, indexes content for non-Russian-language search.

Origins and a company split in two

Yandex unveiled its search engine on September 23, 1997, at the SofTool exhibition in Moscow. Its founders were Arkady Volozh and Ilya Segalovich. Within about five years it added mail, news and images, and launched Yandex.Direct, a contextual advertising service for businesses in Russia.

The crawler's later history is bound up with politics. On May 16, 2017, Ukraine blocked "all of the services of search engine Yandex" alongside VK and Mail.ru sites, according to Meduza. After Russia's full-scale invasion of Ukraine in 2022, the company sold its News and Zen products and the yandex.ru homepage to VK, completing the deal on September 12, 2022. Volozh was placed under EU sanctions that year and stepped down.

On January 25, 2023, 44.7GB of internal source code appeared online. Yandex said a former employee had leaked fragments of an old repository; according to Ars Technica, the files listed 1,922 ranking factors.

The structural break came in 2024. Dutch-registered Yandex N.V. agreed to sell its Russia-based businesses on February 5, 2024, for RUB 475 billion, a price that reflected a 50% discount mandated by Moscow. The EU lifted sanctions on Volozh in March 2024. The second stage closed on July 15, 2024, according to Interfax, and the Yandex brand stayed with the Russian buyers. The Dutch parent renamed itself Nebius Group in August 2024. Dollar conversions of the deal vary between reports, from $5.2 billion to $6.2 billion, because part was paid in shares. YandexBot today belongs to the Russian company that kept the name.

Why it matters to marketers

For brands selling into Russian-speaking markets, YandexBot is the gatekeeper to organic traffic. Statcounter put Yandex at 70.33% of Russian searches in September 2026, against 28.12% for Google. That compares with 71.3% in January 2024, cited in PPC Land coverage of the divestment. Globally, Cloudflare data placed Yandex at 2% of search engine share in December 2025.

Yandex is also one of the engines that pushed IndexNow, the protocol it launched with Microsoft in October 2021 to let sites notify engines of changed URLs instead of waiting for a crawl. By October 2023 participation had reached 60 million websites and 1.4 billion URLs submitted daily, with Seznam.cz and Naver also on board. Google has stayed outside the protocol, so a single IndexNow ping reaches Yandex and bingbot but not Googlebot.

For advertisers, the robots that ignore robots.txt matter most. A landing page blocked to all Yandex robots will still be visited by YaDirectFetcher if it receives Yandex Direct traffic.

Limitations and disputes

The crawler's appetite has grown faster than the traffic it returns. Cloudflare measured Yandex's crawl-to-referral ratio rising 38.3% between January and July 2025, from 15.5 to 21.4 crawls per referral, while Google's settled at 5.4 by July.

Data jurisdiction is a second concern. The Dutch Data Protection Authority fined MLU B.V., parent of ride-hailing service Yango, EUR 100 million on April 1, 2026 for transferring Finnish and Norwegian user data to Yandex.Taxi LLC and Yandex LLC in Russia. The case did not concern the crawler, but it shows how European regulators treat data flows to Yandex entities.

Spoofing is the third. Scrapers routinely borrow the YandexBot name, which makes DNS verification a necessity rather than a precaution. And the long list of robots exempt from general rules means a blanket "User-agent: Yandex" block does not deliver a complete opt-out.

Not the same as

Yandex (the token) is the robots.txt name covering every Yandex robot; YandexBot covers only the main indexer.

YandexAdditional governs use in AI answers, much as Google-Extended does for Gemini. Unlike Google-Extended, it is a named user agent, not purely a control token. Googlebot, by contrast, remains a mixed-use crawler feeding search and AI from one fetch.

YandexComBot indexes for non-Russian-language search under a different domain.

Yandex Direct robots check ad pages and are not part of indexing for organic results.

Recent developments

On April 7, 2026, Yandex added an "Alice AI" tab that switches between classic results and a dialogue with its Alice AI LLM, according to Russian outlet www1.ru, with source links kept beside each answer. That raises the stakes of the YandexAdditional opt-out. Meanwhile automated traffic keeps rising: Cloudflare Radar showed bots at 57.4% of HTML traffic in the week to June 5, 2026, with search-related bots at 10.7%. A practitioner list of user agent strings circulated in June 2026 underlined how many crawlers site owners now have to recognise by name.

Timeline

  • September 23, 1997: Yandex unveils its search engine at the SofTool exhibition in Moscow
  • May 16, 2017: Ukraine blocks all Yandex services by presidential decree
  • February 22, 2018: Yandex stops honouring Crawl-delay in robots.txt
  • 2018: Host directive retired in favour of redirects and canonical tags (date per third-party documentation)
  • October 2021: IndexNow launched by Microsoft and Yandex
  • 2022: Arkady Volozh placed under EU sanctions
  • September 12, 2022: News, Zen and the yandex.ru homepage transfer to VK
  • September 2022: RFC 9309 formalises robots.txt without Crawl-delay
  • January 25, 2023: 44.7GB of Yandex source code leaked, revealing 1,922 ranking factors
  • October 2023: IndexNow reaches 60 million websites and 1.4 billion daily URL submissions
  • February 5, 2024: Yandex N.V. agrees to sell Russian businesses for RUB 475 billion
  • March 2024: EU lifts sanctions on Volozh
  • July 15, 2024: Second stage of the sale closes
  • August 2024: Yandex N.V. renamed Nebius Group
  • January to July 2025: Yandex crawl-to-referral ratio rises from 15.5 to 21.4
  • April 1, 2026: Dutch DPA fines Yango parent MLU B.V. EUR 100 million over transfers to Yandex entities
  • April 7, 2026: Alice AI dialogue tab added to Yandex search
  • September 2026: Statcounter puts Yandex at 70.33% of Russian search

Summary

Who: YandexBot is operated by the Russian company that kept the Yandex brand after Yandex N.V. sold its Russia-based businesses in 2024; the Dutch former parent now trades as Nebius Group. Site owners, SEO teams and advertisers targeting Russian-speaking markets manage its access.

What: The main indexing crawler for Yandex search, identified by the YandexBot/3.0 user agent and verifiable through reverse and forward DNS lookups on yandex.ru, yandex.net or yandex.com hosts. It reads robots.txt, supports the Yandex-specific Clean-param directive, ignores Crawl-delay, and sits within a family of more than 40 Yandex robots.

When: Yandex search launched on September 23, 1997. Crawl-delay support ended on February 22, 2018, IndexNow launched in October 2021, and the company split closed in July 2024.

Where: Primarily Russia, where Yandex held 70.33% of searches in September 2026, and neighbouring markets; its documentation sits in the Yandex Webmaster help centre.

Why: Visibility in Russia's dominant search engine depends on YandexBot fetching a page. Its distinct robots.txt dialect, its AI opt-out agent and its ownership history make it a separate case from Googlebot and bingbot rather than a copy of them.