Bingbot is the automated program Microsoft uses to fetch pages from the public web so they can be indexed, ranked and served in Bing search results and in the answers Microsoft's AI systems generate. It exists for the plainest of reasons: an index can only hold documents that something went out and retrieved. Its reach extends well past Bing's own results page. Yahoo Search runs on Bing's index, DuckDuckGo draws on it, Copilot grounds answers in it, and Ecosia sourced results from it for years. One robots.txt line addressed to bingbot governs visibility across all of them at once.

Microsoft writes the name in lower case inside the user agent string and the robots.txt token, though its own blog posts and most trade coverage capitalise it. Robots.txt matching is case-insensitive, so the variation carries no technical consequence.

The strings a server sees

Bingbot announces itself through two production user agent strings. The desktop variant reads Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/W.X.Y.Z Safari/537.36. The mobile variant wraps the same bingbot/2.0 marker inside an Android 6.0.1 Nexus 5X descriptor. W.X.Y.Z is replaced by the Microsoft Edge build in use at the time. A practitioner reference published in June 2026 recorded the live value as Chrome/116.0.1938.76. That token is absent from the OpenAI and Anthropic strings, so a filter matching on the word Chrome catches bingbot and misses GPTBot. Both variants answer to the single robots.txt token bingbot; no directive separates desktop from mobile fetching.

How a crawl proceeds

Discovery runs on three inputs: links on known pages, XML sitemaps, and direct submission. Before fetching, bingbot requests robots.txt and applies whatever group targets its token, falling back to the wildcard group when none exists.

Fetched HTML then passes through a rendering step. Microsoft announced in October 2019 that bingbot would adopt Chromium-based Microsoft Edge as its JavaScript engine, committing to track the current stable release rather than freeze on a fixed version, and migrated sites in batches over the following months. A page rendering correctly in a recent Edge or Chrome build will therefore generally render correctly for bingbot, provided the scripts and stylesheets it depends on are not themselves disallowed.

Bing supplies a robots.txt tester, launched in March 2022, and a URL Inspection tool, successor to Fetch as Bingbot, returning the HTML the crawler actually received alongside the last crawl timestamp.

Verification and rate control

User agent headers are trivially forged, so Microsoft publishes two verification routes. The older is a reverse DNS lookup resolving to a hostname ending in search.msn.com, in the pattern msnbot-157-55-33-18.search.msn.com, followed by a forward lookup returning the original address. The second is a JSON file of crawler IP ranges at www.bing.com/toolbox/bingbot.json, published by Fabrice Canel on November 15, 2021, days after Google released an equivalent file. That reversed a position Bing stated explicitly in an August 2012 post, which said the company did not publish crawl IP addresses or ranges. The older text has not been withdrawn.

Rate is controlled two ways. Bingbot honours the non-standard crawl-delay directive, accepting whole numbers from 1 to 30, each mapping to seconds between requests, so a value of 30 permits one request every half minute. Crawl Control inside Bing Webmaster Tools presents an hourly grid across the week, shifting intensity away from peak traffic periods. Microsoft has stated that crawl-delay takes precedence when both are set, and calls its default self-regulating behaviour crawl politeness. Googlebot ignores crawl-delay entirely, one of the few genuine points of divergence between the two crawlers.

Push instead of pull

Microsoft has spent years reducing how much crawling bingbot needs to do. In January 2020 it piloted a content submission API with Botify, letting enterprise sites push HTML directly and bypass the 10,000 URL ceiling then imposed by the Bing API. "Instead of frequently crawling websites to check for new and updated content, webmasters can now get their content indexed in real-time," Canel said at the time.

That work became IndexNow, the open protocol Microsoft launched with Yandex in October 2021, which notifies participating engines the moment a URL is added, changed or removed. By October 2023 it covered 60 million websites and 1.4 billion URLs submitted daily, and Microsoft issued sector-specific cadence guidance in September 2024. Google has never adopted it, an absence PPC Land examined in December 2024 as a difference in indexing philosophy rather than a technical obstacle. On July 31, 2025, Canel and Krishna Madhavan repositioned sitemaps as critical infrastructurerather than an optional aid, arguing that AI-assisted retrieval leans harder on structured signals such as lastmod accuracy than brute-force crawling can support.

From msnbot to bingbot

Microsoft began prototyping its own crawler in 2003 under the name msnbot, ending a dependence on third-party index suppliers. MSN Search entered beta in 2004 and launched fully on February 1, 2005, with later crawler versions adding HTTP compression and conditional requests. The engine was renamed twice, to Live Search and then to Bing in June 2009, while the crawler kept its old name; the mismatch was corrected in October 2010, when bingbot replaced msnbot. The 2019 move to evergreen Edge rendering was the next structural change. In April 2022 Microsoft split the single legacy string into the current desktop and mobile pair, completing the transition that autumn. The Edg token present in the December 2019 preview strings did not survive into production.

Why it matters to marketers

Microsoft's search and news advertising revenue reached $13.9 billion in fiscal year 2025. Log analysis by Botify and Nectiv, reported in April 2026, recorded 5.49 billion bingbot events in a 30-day window against 18.2 billion for Googlebot, placing bingbot at roughly 30% of Google's crawl volume and OpenAI's combined crawlers at about 14% the size of Bing's.

The index that bingbot builds now feeds two distinct consumers. Microsoft set out the divergence on May 6, 2026, when Madhavan, Knut Risvik and Meenaz Merchant argued that an index built for AI grounding differs from a ranking index across five dimensions: factual fidelity, source attribution quality, freshness, coverage of high-value facts, and contradiction handling. Risvik followed on June 2, 2026 with Web IQ, a grounding API suite returning passage-level evidence at sub-165ms p95 latency, described as a search engine for AI systems rather than people.

Publishers gained measurement in parallel. The AI Performance dashboard entered public preview on February 10, 2026and gained query-to-page mapping on March 23, 2026. Element-level control arrived earlier, with data-nosnippet on October 15, 2025, suppressing marked content from snippets and AI answers while leaving it indexed.

Limitations and disputes

The central objection is that bingbot offers no separation between search indexing and AI grounding. Blocking the token removes a site from Bing, Yahoo and Copilot at once. Data-nosnippet trims what is shown, not what is collected, and no Bing equivalent of Google-Extended exists.

Infrastructure is imposing distinctions Microsoft has not. Cloudflare's July 1, 2026 policy classifies crawlers by behaviour rather than operator and, from September 15, 2026, blocks Training and Agent crawlers by default on advertising-carrying pages for domains newly joining its network. BingBot is named alongside Googlebot and Applebot as a multi-purpose crawler judged by its most restrictive behaviour, so customers who blocked training block Bing's search crawling too. IAB Australia took the opposite view in guidance dated July 22, 2026, placing bingbot in Category A, the discovery and search indexers it likens to a librarian who returns readers to the shelf.

Blocking carries measurable cost. Research from Rutgers Business School and The Wharton School found news publishers who blocked large language model crawlers lost roughly 7% of weekly traffic within six weeks on human panel data, with SimilarWeb figures showing a 23.1% decline. Cloudflare measured search crawlers at 10.7% of HTML traffic across its network by early June 2026, against 50.6% for training bots.

Access to the index itself has narrowed. Microsoft retired the Bing Search and Bing Custom Search APIs on August 11, 2025, directing developers to Grounding with Bing Search in Azure AI Agents at a cost PPC Land calculated as 40% to 483% higher. Earlier price rises had already pushed Qwant and Ecosia to build the Staan index through European Search Perspective, which began serving French queries on August 7, 2025. France's competition regulator held in December 2025 that Qwant was not economically dependent on Microsoft, a conclusion European operators contest. Canel, who shaped Bing's crawler and webmaster tooling across nearly three decades, retired on July 1, 2026.

Disambiguation

Bingbot and AdIdxBot are separate crawlers. AdIdxBot fetches advertising landing pages for quality checks in Microsoft Advertising; its output never reaches organic results.

BingPreview and MicrosoftPreview generate page snapshots for search results and other Microsoft surfaces. Neither builds the index.

Msnbot is the retired predecessor, though stray msnbot/2.0b requests still surface in server logs.

IndexNow is a notification protocol, not a crawler. It tells Bing a URL changed; bingbot still performs the fetch.

Recent developments

On February 5, 2026, Canel and Google's John Mueller jointly discouraged separate markdown pages built for AI crawlers, answering a question from Lily Ray of Amsive. "Really want to double crawl load? We'll crawl anyway to check similarity," Canel wrote, adding that non-user versions of pages are frequently neglected and broken. The exchange placed the practice close to longstanding cloaking policy.

Microsoft shipped a one-click AI toggle for Bing on June 6, 2026, a browser extension suppressing AI summaries, released without regulatory compulsion of the kind the UK imposed on Google three days earlier. Cloudflare's August 21, 2026 transparency conditions for mixed-purpose bot verification apply to any operator combining search and training behaviour, a category Microsoft's single-token architecture places bingbot in.

Timeline

  • 2003: Microsoft begins prototyping msnbot to build its own web index
  • 2004: MSN Search enters beta using msnbot-crawled data
  • February 1, 2005: MSN Search launches publicly
  • June 2009: Microsoft rebrands its search engine as Bing
  • October 2010: Bingbot formally replaces msnbot
  • August 2012: Bing documents reverse DNS verification and states it does not publish crawl IP ranges
  • October 2019: Microsoft announces evergreen rendering using Chromium-based Edge
  • December 2019: Future bingbot user agent strings published
  • January 2020: Content submission API piloted with Botify
  • October 2021: IndexNow launched with Yandex
  • November 15, 2021: Bingbot IP ranges published as a JSON file
  • March 2022: Bing releases a robots.txt tester tool
  • April 2022: Desktop and mobile user agent strings split; migration completes that autumn
  • November 2021 to 2024: Bing API price increases push European engines toward independent indexing
  • October 20, 2024: Bing Webmaster Tools extends search performance history to 16 months
  • February 1, 2025: MSN Search anniversary passes with msnbot long retired
  • July 31, 2025: Microsoft repositions sitemaps as critical AI-era infrastructure
  • August 7, 2025: Ecosia begins serving French queries from the Staan index
  • August 11, 2025: Bing Search and Bing Custom Search APIs retired
  • October 15, 2025: data-nosnippet attribute introduced for Bing
  • February 5, 2026: Canel and Mueller discourage AI-specific markdown pages
  • February 10, 2026: AI Performance dashboard enters public preview
  • March 23, 2026: Grounding query to page mapping rolls out worldwide
  • May 6, 2026: Microsoft publishes its AI grounding index architecture post
  • June 2, 2026: Web IQ grounding APIs launched
  • June 6, 2026: One-click AI search toggle released for Bing
  • July 1, 2026: Fabrice Canel retires from Microsoft
  • September 15, 2026: Cloudflare default blocking of Training and Agent crawlers begins for new domains

Summary

Who: Microsoft operates bingbot, with Fabrice Canel the principal product manager most publicly associated with it until his retirement on July 1, 2026, and Krishna Madhavan, Knut Risvik and Meenaz Merchant now fronting its indexing documentation. Site owners, publishers and technical SEO practitioners manage its access through robots.txt, Crawl Control and Bing Webmaster Tools.

What: A crawler that fetches pages over HTTP, renders them in a Chromium-based Microsoft Edge engine tracking the current stable release, and supplies the index behind Bing Search, Yahoo, DuckDuckGo, Copilot and the Web IQ grounding APIs. It identifies itself with a bingbot/2.0 marker in desktop and mobile user agent strings, honours crawl-delay values from 1 to 30 seconds, and can be verified by reverse DNS to search.msn.com or against a published JSON file of IP ranges.

When: Descended from msnbot, prototyped in 2003 and public from February 2005, renamed bingbot in October 2010, moved to evergreen Edge rendering from October 2019, and split into separate desktop and mobile user agent strings in April 2022.

Where: Documentation, the Verify Bingbot tool and the IP range file sit on Bing's webmaster properties, with technical announcements published on the Microsoft Bing Blogs.

Why: Bingbot is the sole retrieval path into an index that serves several search front ends and Microsoft's AI answering systems simultaneously, and Microsoft offers no token separating those uses. That single point of control is what makes the crawler consequential for publishers, and what places it inside a wider dispute about whether search crawling and AI grounding should be governed by the same rule.