Microsoft today added an AI Scrape-to-Referral Ratio card to Clarity's Bot Analytics dashboard, giving site owners a single number for how much content an AI operator extracts against how many visitors it sends back - and a one-click path from that number into the session recordings of the people who actually arrived.

The release, published on the Microsoft Clarity blog on August 13, 2026 under a Clarity Staff byline, joins two datasets that until now sat on opposite sides of the platform. Bot Analytics counted crawler requests. Clarity's behavioural analytics counted human sessions. The new card divides one by the other.

the PPC Land daily grid

Think you know ad tech? Prove it. PPC Land now runs a daily word game built entirely from the language of programmatic - sixteen terms, four hidden groups of four, one fresh grid every morning. Some tiles look like they belong somewhere they don't, and that misdirection is the whole puzzle. There's a weekly crossword too, drawn from the terminology that fills briefs, DSP dashboards, and measurement decks. Free to play, no account needed. Find out whether you really know your bid shading from your supply path optimization.

Learn more

According to Microsoft, the card is designed to let site operators evaluate the tradeoff between content extraction and traffic return, then investigate how AI-referred visitors behave once they arrive. The framing in the announcement is blunt about the problem it addresses: the challenge for site owners is no longer just tracking which AI bots scrape their pages, but understanding whether that activity leads to real customer visits and engaged user behaviour.

What the card displays

The illustration accompanying Microsoft's post shows an average scrape-to-referral ratio of 6,000:1 against a total of 41 referrals. That figure is a product screenshot rather than a published benchmark, and Microsoft does not present it as a cross-site average. It does, however, indicate the order of magnitude the card is built to display. A ratio in the thousands is not an edge case in this category of measurement. It is the ordinary reading.

Six changes ship together, according to the announcement.

The AI Scrape-to-Referral Ratio card compares scrape activity against referral traffic in a single metric, which Microsoft describes as a way to understand whether AI platforms are returning value in the form of visits rather than only extracting content.

An operator-level breakdown presents a ranked list of referring operators. The stated purpose is identifying which AI sources send traffic back and which show high scrape activity with limited referral return. Ranking matters more than the headline average here. A site-wide ratio of 6,000:1 tells an operator very little if one crawler accounts for nearly all requests and another accounts for nearly all referrals.

Mapped-domain coverage safeguards restrict the calculation to correctly mapped domains. Microsoft states this prevents misleading rollups when CDN and Clarity coverage do not fully align - a technical caveat that carries real weight, because the two halves of the ratio come from different collection layers. Crawler requests are read from server logs delivered through a content delivery network. Referral sessions are read from Clarity's JavaScript tag. If those two layers cover different sets of domains, the resulting quotient is arithmetic without meaning.

Clearer handling of coverage gaps extends the same logic. When bot data and referral data cover different domains, Clarity now supplies context so that site owners understand how domain coverage affects the ratio.

A direct path to session recordings lets users click from referral insights into the Recordings tab with filters already applied for the selected AI referral source. Microsoft says this makes it easier to validate attribution and inspect real user journeys.

Finally, better traffic quality analysis moves the assessment beyond top-line referral counts, allowing comparison of whether AI-referred visitors scroll, engage, convert, or leave quickly.

The onboarding sequence Microsoft publishes runs through six steps: open Bot Analytics in the AI Visibility Dashboard, review the ratio card, check coverage context to confirm mapped domains are aligned, select a referral source to open filtered recordings, watch those recordings for engagement patterns and actions such as purchases, sign-ups or form fills, and use the result to compare AI sources.

The ratio was somebody else's metric first

The scrape-to-referral ratio is not a Microsoft invention, and the number it produces has been politically loaded since it entered circulation.

Cloudflare published the first widely circulated crawl-to-refer analysis on August 29, 2025, showing one operator's ratio reaching 286,930 crawls per referral in January 2025 before falling to 38,000 by July of that year, while another held near 1,091 across the same stretch. A December 2025 analysis of robots.txt files across 3,879 of the top 10,000 domains put Anthropic, OpenAI and Perplexity in a band running from 10:1 to over 100:1 in that dataset, with AI bots originating 4.2% of HTML requests during a year when global internet traffic grew 19%.

On July 1, 2026, Cloudflare turned the metric into a product. Its Attribution Business Insights dashboard surfaced per-operator ratios spanning 118 crawls per referral at the low end to nearly 50,000 at the high end, calculated against referrals tracked through UTM parameters and viewable across 24-hour, 7-day and 30-day windows.

Microsoft is therefore shipping into an established measurement category rather than opening one. The differences are architectural. Cloudflare sees the crawl because it sits in the request path. Clarity sees the session because it sits in the page. Neither sees both natively, which is why Clarity's version requires a CDN connection and why Microsoft has built explicit safeguards around domains where the two data sources do not overlap.

The second difference is what happens after the number appears. Cloudflare's dashboard was built, according to its own announcement, for business decision-makers weighing access policy. Clarity's card routes into session replay. That is a different question being asked of the same arithmetic: not whether to permit a crawler, but whether the visitors it eventually sends are worth anything.

Volume was never the interesting part

The case for looking past referral counts has been building in the data for eighteen months, and it cuts in both directions.

AI referral volume remains small. Research published in February 2026 found ChatGPT sends roughly 190 times less traffic to websites than Google despite handling about 12% of its query volume. Clarity's own research team, in a study of more than 1,200 publisher sites published on December 18, 2025, found AI-referred traffic had grown 155% over eight months while still accounting for under 1% of total visitors in the dataset.

Quality has been the counter-argument. The same Clarity research recorded AI-referred visitors converting to sign-ups at 1.66% against 0.15% for search, 0.13% for direct and 0.46% for social. Clarity has also documented that AI-referred visitors tend to land on deep pages rather than homepages, entering mid-funnel and producing engagement signals that do not resemble organic search behaviour.

Other vendors report a wider spread. Criteo's internal data from a sample of 500 retailers in February 2026 placed LLM-referred conversion at roughly 1.5 times other referral channels, rising in May 2026 to nearly twice traditional search in consumer electronics, lifestyle and home categories. Invoca's benchmark report of July 14, 2026 found ChatGPT-referred calls converting to leads at 49%, the highest rate of any channel it measured. Against those figures sits a study of 973 e-commerce sites with $20 billion in combined revenue that found ChatGPT referrals underperforming traditional channels on conversion and revenue per session.

The variance is the point. Conversion multiples for AI traffic range from below parity to eleven times search depending on vertical, sample and definition, which makes any single published benchmark close to useless for an individual site owner. Session recordings filtered by AI source address exactly that gap. They replace an industry average with an observation of a specific site's own visitors.

The attribution problem the card inherits

One structural weakness sits underneath the referral half of the ratio, and Microsoft's announcement does not claim to solve it.

AI referral traffic is systematically undercounted. Google confirmed on May 23, 2025 that noreferrer elements in AI Mode link code were stripping referrer values, causing those clicks to register as direct traffic. OpenAI added UTM parameters to ChatGPT links in June 2025. Neither fix was complete. The Gemini iOS app was found in October 2025 to use an undocumented user agent string. Google Analytics only added a native AI Assistant channel grouping on May 13, 2026, and a Source Group field to normalise variant source strings followed in June.

A ratio built on an undercounted denominator overstates the imbalance. If a meaningful share of AI-referred sessions arrive without an identifiable origin, the referral count is low for measurement reasons rather than economic ones, and the ratio reads worse than the underlying exchange actually is. The click-through into session recordings partly compensates, since a recording of an engaged visitor exists regardless of how that visitor was labelled, but the top-line number retains the bias.

The crawl half carries the mirror problem. Reporting on PatronView's decision to block Amazon's crawler after 117,000 daily page reads documented how JavaScript-based visitor tracking systematically undercounts traffic that actually reaches a server. Clarity's use of CDN logs for the bot side avoids that specific distortion, which is precisely why the CDN connection is not optional.

What it costs and what it requires

Nothing, and a CDN.

Microsoft has held the free-with-no-traffic-limits position across the entire AI visibility sequence, giving away tooling that rivals charge for. Cloudflare's comparable dashboard reached Bot Management customers, with its supporting documentation specifying the Enterprise tier. Topic Insights, Query Topics, Citations and Bot Analytics all shipped inside Clarity at no additional cost.

The gating requirement is infrastructural rather than commercial. When Clarity added robots.txt violation detection on June 23, 2026, Microsoft specified that project administrators must connect a supported CDN through the AI Visibility section in Project Settings, naming Fastly, Amazon CloudFront and Cloudflare, with WordPress sites covered automatically through the latest Clarity plugin. That dependency draws a line through the addressable audience. Sites without one of those three providers, and without WordPress, cannot populate the bot side of the ratio at all.

A release every fortnight

Today's card is the fourth AI visibility addition to Clarity in roughly five weeks, and the seventh distinct release in the sequence since January.

Bot Activity tracking arrived on January 21, 2026, exposing which crawlers requested content and at what volume through server-side log collection. Citations reached general availability on May 13, 2026, bringing grounding queries, cited pages, share of authority and AI referral trends to every project. Web IQ and the wider Microsoft AI advertising bundle landed on June 17 ahead of Cannes Lions, with the company stating that AI traffic is growing eight times faster than human traffic. Robots.txt violations followed on June 23. Topic Insights shipped July 9. Query Topics entered beta on July 22, grouping citation grounding queries into ranked topics with export support. Earlier this month Clarity split citation records by branded and non-branded queries.

Read as a sequence, the releases trace a lifecycle. Bot Activity showed the request. Citations showed whether the request produced a mention. Topic Insights and Query Topics showed which subject areas those mentions clustered around. Branded segmentation showed whether the mention required prior awareness. Today's card closes the loop by asking what the whole chain delivered in visitors, and whether those visitors did anything.

Why this matters for marketing teams

Measurement infrastructure for AI visibility is arriving faster than agreement on what it should measure. The IAB published a framework on August 4, 2026 noting that only 16% of brands systematically track AI visibility, and identifying the absence of a common definition of a citation as part of the reason. Two members of the working group came from Microsoft Clarity.

For publishers, the scrape-to-referral ratio is an input to an access decision with a deadline attached. Cloudflare has set September 15, 2026 as the date after which Training and Agent crawlers are blocked by default on advertising-carrying pages for domains newly joining its network. Research from Rutgers Business School and The Wharton School found news publishers who blocked AI crawlers through robots.txt lost 23.1% of monthly visits without achieving proportional protection. A per-operator ratio makes that tradeoff site-specific rather than ideological: an operator returning visitors at a survivable rate is a different proposition from one that has never sent a reader.

For brand and performance teams, the value sits in the recordings rather than the ratio. Budget allocation toward answer engine optimisation currently rests on visibility scores and citation counts, metrics that a Similarweb framework found change roughly 50% monthly with only 11% overlap across platforms. Watching what an AI-referred visitor does on a specific site produces a smaller sample but a more defensible one. Whether an operator's visitors complete a form is a question with a local answer.

The unresolved element is comparability. Clarity's ratio, Cloudflare's ratio and the figures publishers compute themselves are not calculated on identical inputs, and Microsoft's mapped-domain safeguards exist because even a single platform's internal inputs do not always align. Ratios that cannot be compared across sites cannot be aggregated into the industry benchmark the IAB framework says is missing.

Timeline

Summary

Who: Microsoft, through the Clarity analytics team, publishing on the Clarity blog under a Clarity Staff byline. The release affects publishers, brands, content teams and marketers measuring how AI crawling activity translates into website visitors.

What: An AI Scrape-to-Referral Ratio card added to Clarity's Bot Analytics dashboard, comparing AI scrape activity against referral traffic in a single metric. The release also includes an operator-level ranked breakdown of referring AI sources, mapped-domain coverage safeguards limiting the calculation to correctly mapped domains, contextual handling of coverage gaps between bot and referral data, a direct click-through into session recordings with filters pre-applied for the selected AI referral source, and traffic quality analysis extending beyond top-line referral counts to scroll, engagement and conversion behaviour. Microsoft's illustration displays an average ratio of 6,000:1 against 41 total referrals.

When: Announced August 13, 2026, roughly a week after Clarity split citation records into branded and non-branded queries, and six weeks after Cloudflare opened a competing per-operator ratio dashboard on July 1, 2026.

Where: Inside the Bot Analytics dashboard within Microsoft Clarity's AI Visibility Dashboard, with the session recordings path routing into Clarity's Recordings tab. Bot data collection depends on a connected content delivery network, with Fastly, Amazon CloudFront and Cloudflare named as supported providers when the earlier violations feature shipped, alongside automatic coverage for WordPress sites running the current Clarity plugin.

Why: Crawler volume and referral volume have been measured separately, leaving site owners unable to judge whether extraction produces any traffic return, or whether the traffic that does arrive has value. Clarity's own research found AI referrals growing 155% over eight months while remaining under 1% of visitors, and converting at 1.66% against 0.15% for search. Published conversion multiples for AI traffic vary widely by vertical and sample, which makes site-specific observation more useful than industry averages. Joining bot data to session replay lets operators test both halves of the exchange against their own visitors rather than a benchmark.