Google users whose searches were switched entirely to AI Mode for a week clicked through to outside websites 18.8 percentage points less often than people using standard Google Search, according to a preregistered field experiment with 1,100 US participants that researchers at the University of Pennsylvania and Northeastern University posted to arXiv on August 18, 2026. The same participants visited news sites, Reddit and Wikipedia less often, rated Google as less trustworthy, useful and satisfying, and were more likely to run searches on Bing, DuckDuckGo or Yahoo. A second condition, which tried to remove AI Overviews from ordinary results, produced an estimated 8.8-point rise in click-through, but that estimate depends on a statistical adjustment the authors had to make after a change to Google's page code broke their filter partway through the study.
In Short
In March 2026, researchers used a browser extension to change what about 1,100 US Google users saw for a week, and those pushed entirely into AI Mode clicked out to websites 18.8 percentage points less often than people on normal Google. This matters for publishers and advertisers because AI Mode also cut visits to news sites, Reddit and Wikipedia, lowered trust in Google, and sent more users to rival search engines, while removing AI Overviews appeared to raise clicks in an estimate weakened by a filter that stopped working mid-study. The paper is a preprint that has not been peer reviewed, it tested a forced switch with a young, left-leaning sample over seven days, and it says AI Mode carried no ads at the time, a statement that sits awkwardly with Google's February 2026 introduction of a sponsored shopping format there. What changes is the evidence base: together with an earlier 2026 experiment by researchers at the Indian School of Business and Carnegie Mellon, there are now two randomized studies pointing to fewer outbound clicks from Google's AI search features.
The paper and its authors
The paper is titled "AI in Search Reduces Publisher Referrals Without Improving User Experience: Experimental Evidence." It carries the identifier arXiv:2608.18352v1 and is filed under Information Retrieval (cs.IR), with cross-listings in Computers and Society and Human-Computer Interaction. Its arXiv page records a single version, submitted on Tuesday, August 18, 2026 at 22:08 UTC by Stephanie Wang, and as of November 17, 2026 no revision had been posted. Every page of the 46-page PDF carries a "Draft" watermark.
Five researchers are listed. Stephanie T. Wang, the corresponding author, and Danaé Metaxa are with the Department of Computer and Information Science at the University of Pennsylvania. Jeffrey Gleason and Christo Wilson are at Northeastern University's Khoury College of Computer Sciences, and Yakov Bart is at Northeastern's D'Amore-McKim School of Business. Wang and Gleason are marked as having contributed equally, and the author contributions section credits the two of them with the software, formal analysis and data curation.
arXiv is a preprint server. Papers posted there are checked for basic suitability but are not peer reviewed, and no journal reference or venue appears on the paper's page. The findings below are therefore the authors' own, as submitted, and have not been through independent review.
The paper discloses its funding and its tools. The work was "supported in part by the National Science Foundation under award IIS-2442711," and Gleason was supported by the NSF's Platforms for Exchange and Allocation of Resources traineeship. "The authors declare that they have no competing interests," according to the paper. It also states that "Generative AI (GPT-5.5-mini) was used to lightly edit and rephrase author-written text for clarity and fluency," and that "All analyses, interpretations, and conclusions were developed and verified by the authors." The Institutional Review Boards of both universities approved the study. The authors preregistered their hypotheses on the Open Science Framework and say aggregated data and code are on Harvard Dataverse.
What the researchers set out to test
The question at the center of the paper is one that publishers and Google have argued over since AI Overviews arrived in May 2024: do AI answers on the results page send people onward to websites, or do they replace the visit? The authors frame the dispute directly. Publishers "view AIO and AI Mode as substitutes for their content and report falling referral traffic from Google Search," while "Google disputes these claims, arguing that AI features are complements that have kept overall referral traffic steady, improved click quality, and expanded opportunities for websites to be surfaced through longer, more complex queries."
Google's position has been on the record for some time. In August 2025, Google's head of Search, Liz Reid, said total organic click volume had stayed relatively stable year over year and that Google was sending slightly more quality clicks to websites than a year earlier. Senior vice president Nick Fox wrote on July 17, 2026 that Google was sending billions of clicks to websites every week through AI features in Search, without giving a denominator.
The researchers preregistered three primary hypotheses. The first predicted that AI Mode would lower trust in Google's results and that removing AI features would raise it. The second predicted that AI Mode would lower click-through to external sites and that removing AI features would raise it, based on publisher reports and observational data. The third, built on Google's own claims that AI features increase engagement, predicted that AI Mode would increase the number of search sessions. Four secondary hypotheses covered perceived usefulness, satisfaction, agency and personalization; clicks to news sites, Wikipedia, Reddit and ads; time and searches per session; and switching to other search engines.
How the experiment worked
According to the paper's methods section and supplementary materials, the study ran in eight steps.
- Screening. Participants confirmed four statements on the consent page: "I am living in the United States," "I am 18 years old or older," "My main web browser is Google Chrome," and "My main search engine is Google Search."
- Pre-survey. Before anything changed, participants answered questions on their familiarity with and feelings about "LLM applications such as ChatGPT, Claude, and Gemini," and rated how believable, trustworthy, accurate, biased and complete they found information on Google, along with agency, personalization, usefulness and satisfaction. The survey included an attention check ("Please select 'Slightly' for this question").
- Installation. Participants signed in to the study's web application through Google social authentication, then installed a browser extension from the Chrome Web Store. According to the paper, the extension works in "all Chromium-derived browsers, including Google Chrome." The web application was built with Django and ran on a Northeastern University server backed by a PostgreSQL database.
- Baseline. For three days, every participant used unmodified Google Search while the extension logged searches, outbound clicks and copies of the results pages. This produced pre-treatment values for each behavioral outcome.
- Random assignment. After the baseline, each participant was assigned to one of three conditions "with equal probability and no constraints imposed on group balance."
- Treatment. For seven days, the extension enforced the assigned condition inside the browser. In No AI Search, it watched the page for changes and set the display property of any AI Overview container to "none", whether the Overview sat at the top of the page, in the middle or inside a People Also Ask box. It also hid the AI Mode tab and the AI Mode prompt on the Google home page, and redirected AI Mode searches to standard results. The paper notes that "browser extensions cannot hide the 'AI Mode' prompt in the Chrome browser's address bar." In AI Mode Search, the extension redirected every Google search to AI Mode through the URL parameter udm=50 and hid the "All" and "Web" tabs. News, shopping and image searches stayed available, and participants saw a one-time notice that they would be using AI-driven search for the rest of the study. Current Search received no changes.
- Post-survey. After seven days, participants repeated the trust and experience questions. They also compared an AI Mode response with a standard Google response for three news queries, drawn at random from ten trending queries taken from Google Trends the day before the survey went out.
- Debrief. Participants were paid and shown a statement explaining the purpose of the study, the three conditions and why those details had been withheld until the end.
The ten news queries give a sense of the period: "tsa agents government shutdown," "iran israel war ceasefire," "laguardia plane crash," "us treasury insolvency," "us army enlistment age," "meta social media addiction trial," "retirement benefits social security," "no kings protest near me," "covid 19 cicada variant" and "ice at airports."
How the outcomes were defined
The definitions matter for anyone comparing this study with traffic data from analytics tools. The paper defines click-through rate as "The total number of external clicks originating from Google divided by the total number of Google searches. The numerator excludes clicks to google.com (e.g., follow-up searches). The denominator excludes searches on Google's vertical search pages (e.g., images, news, shopping). Each turn of a conversation in AI Mode is counted as a separate search. This is a user-level metric."
That last rule has consequences. A participant who asks AI Mode three follow-up questions has run three searches under this definition, which lowers the click-through rate of a long conversation. The authors tested the point: counting each AI Mode conversation as one search instead of each prompt, the estimated fall in click-through rate is 14.9 points rather than 18.8, and still statistically significant.
Searches were grouped into sessions using a five-minute rule: "if the inter-arrival time between two searches exceeds 5 minutes, the second query is considered a new session." Checked against 100 manually labelled pairs of queries from a pilot, the rule scored 0.84 on the F1 measure. Ad clicks were "identified using the gclid URL parameter," the click identifier that Google Ads attaches to each ad click. News sites were identified using a curated list of news domains, and Reddit and Wikipedia were singled out because they were the only domains to receive at least 5% of clicks from Google during the pilot.
Who took part
Recruitment ran in waves from March 17 to March 19, 2026, through the research platform Prolific and among Northeastern University students in a work-study program with one of the authors. Prolific participants were paid $2 for screening, the first survey and installing the extension, and $8 for finishing the study. Northeastern students received $17 an hour.
| Item | Figure, according to the paper |
|---|---|
| Recruitment dates | March 17 to 19, 2026 |
| Study period per participant | 3 days baseline plus 7 days assigned condition |
| Enrolled | 1,444 (1,387 Prolific, 57 Northeastern) |
| Made at least one search during the 7 days (behavioral sample) | 1,100 |
| Completed the final survey | 956 (343 No AI, 304 Current, 309 AI Mode) |
| Behavioral sample by condition | 386 No AI, 356 Current, 358 AI Mode (PPC Land calculation from the reported sample sizes) |
| Preregistered target | 1,200 participants |
| Pilot used for the power analysis | 72 participants (61 Prolific, 11 Northeastern) |
| Age, behavioral sample | 75% under 45 |
| Education, behavioral sample | 87% with at least some college |
| Party, behavioral sample | 58% Democrat, 20% Republican |
| Gender, behavioral sample | 50% women, 48% men, 2% another gender |
| Baseline searches per day | 9.2 (standard deviation 14.3) |
| Baseline search sessions per day | 4.0 (standard deviation 4.7) |
| Baseline clicks per day | 4.1 (standard deviation 7.0) |
| Share of searches showing an AI Overview | 36% |
| Share of searches in AI Mode before assignment | 0.6% |
Two numbers in that table frame everything that follows. First, AI Mode was almost unused before the experiment began: 0.6% of searches. The AI Mode condition therefore moved people from near-zero use to near-total use overnight, which is not how most people would meet the product. Second, the sample does not look like the US population on age, education or politics. The authors say race, ethnicity and gender were similar to US Census figures.
The researchers checked that the three groups were balanced on demographics, baseline attitudes and baseline behavior, and found no evidence of imbalance (p = 0.305). Dropout was somewhat higher in the AI Mode group: 36.6% did not complete the final survey, against 32.5% for No AI and 32.3% for Current Search, and 26.5% ran no searches at all during the treatment week, against 24.0% and 20.7%. Neither difference was statistically significant (p = 0.292 and p = 0.109), and the authors use that result to treat their AI Mode estimates as causal.
What AI Mode did to clicks
The AI Mode condition worked largely as designed. During the treatment week, 94.7% of searches in that group were routed to AI Mode. Six participants reached ordinary web results through a non-standard URL on which the extension did not fire.
The headline estimates come from comparing everyone assigned to AI Mode with everyone assigned to unmodified Google, an intent-to-treat comparison. The paper reports them in percentage points for rates and shares, in raw units for counts and minutes, and in points on seven-point scales or standard deviations for survey measures.
| Behavioral outcome | AI Mode vs Current Search (intent to treat) | Statistically significant? |
|---|---|---|
| Click-through rate to external sites | -18.8 points (95% CI -22.2 to -15.3) | Yes |
| Click-through rate, one conversation counted as one search | -14.9 points | Yes |
| External clicks per day | -2.27 | Yes |
| Search sessions per day | -0.92 (95% CI -1.30 to -0.55) | Yes, opposite to the preregistered prediction |
| Share of users clicking any news site | -12.5 points | Yes |
| Share of users clicking Reddit | -21.2 points | Yes |
| Share of users clicking Wikipedia | -9.9 points | Yes |
| Share of users clicking an ad (gclid) | -42.7 points | Yes; the authors attribute it to AI Mode showing no ads |
| Minutes per session | +0.43 | Yes |
| Searches per session | +0.14 | No |
| Share of searches phrased as questions | +0.004 | No |
| Share of users searching on Bing, DuckDuckGo or Yahoo | +11.2 points | Yes |
The pattern is consistent. People in AI Mode stayed longer in each session, by about 26 seconds on PPC Land's conversion of 0.43 minutes, but ran fewer sessions and left Google less often. Before assignment, the AI Mode group had averaged 3.33 clicks a day in the paper's balance table, so by a PPC Land calculation the estimated drop of 2.27 clicks a day is equal to about two-thirds of that starting level. The paper does not give the average click-through rate of the control group during the treatment week, so the relative size of the 18.8-point fall cannot be read directly from it.
The fall in news clicks survived both of the authors' robustness checks. Using Google's own Topics API to classify news domains instead of the curated list, the AI Mode effect on the share of users clicking news sites was smaller, at 8.1 points, but still significant. Results also held when clicks were counted per day rather than as a yes-or-no measure.
One result ran against expectations. The authors had predicted, from Google's statements that AI features increase search engagement, that AI Mode would increase the number of search sessions. It did the opposite, cutting sessions by 0.92 a day. The effect was strongest among heavy users: for participants who had run more than four sessions a day before the experiment, AI Mode reduced daily sessions by a further 2.01 (p = 0.005), the only subgroup effect that remained significant after correction for multiple comparisons. Familiarity with chatbots and age made no significant difference.
Removing AI Overviews: a smaller and shakier signal
The No AI condition is where the study ran into trouble. On the first day of treatment, the extension hid about 90% of AI Overviews. By the tenth day of the study, it was hiding close to none. According to the paper, "The cause of this breakdown was a change to AI Overview HTML rolled out by Google during the study period that broke our extension's logic for identifying and hiding AI Overviews." Across the week, 51.1% of AI Overviews were hidden, and the median participant had half of them removed.
The preregistration had anticipated non-compliance, defining a compliant participant as one who received the assigned treatment on at least 90% of searches. That threshold became unworkable, so the authors measured compliance as a continuous share of Overviews hidden and estimated a local average treatment effect using two-stage least squares, with assignment as the instrument. The paper states plainly that "Our pre-registration did not specify how we would estimate LATEs in the case of non-compliance." It also excluded, without preregistration, users who never ran a search that would have triggered an AI Overview: 17.9% of the No AI group and 16.3% of the Current Search group.
The difference between the two estimates is large.
| No AI Search vs Current Search | Intent to treat (all assigned) | Adjusted for compliance (LATE) |
|---|---|---|
| Click-through rate | +2.7 points, not significant (p = 0.124) | +8.8 points (95% CI 2.3 to 15.3), significant |
| Share of users clicking news sites | +7.1 points, significant (p = 0.033) | +20.2 points, significant |
| Share clicking Reddit | -1.6 points, not significant | -4.9 points, not significant |
| Share clicking Wikipedia | +2.4 points, not significant | +5.9 points, not significant |
| Share clicking an ad | +2.4 points, not significant | +9.8 points, not significant |
| Search sessions per day | +0.10, not significant | +0.35, not significant |
| Minutes per session | -0.22, significant before adjustment only | -0.59, significant |
Read plainly, simply assigning people to a version of Google with AI Overviews partly removed did not produce a statistically significant change in click-through. The 8.8-point figure is an estimate of what full removal would have done, scaled up from partial removal, and it holds only if, in the paper's words, "assignment only affected outcomes through AI Overview exposure." The authors add: "If the interface change altered participants' experience more broadly, that assumption may not hold." The news-site effect in the No AI condition did not survive either robustness check; using the Topics API classification, it fell to roughly zero.
The study does not show trust or satisfaction changing when AI Overviews were removed. "Removing AI Overviews produced no detectable change in perceived trust in information found on Google, nor on measures of search experience," according to the discussion.
Trust, satisfaction and the drift to Bing
The survey results for AI Mode were uniformly negative. Trust in information found on Google, measured on a five-item scale with a seven-point response range, fell by 0.34 points. On standardized measures, perceived usefulness fell by 0.59 standard deviations, satisfaction by 0.73, agency by 0.66, and the sense that results were "personalized and relevant" by 0.42. The personalization result ran against the preregistered prediction, which had been based on Google's claims about AI-driven personalization.
| Survey outcome | AI Mode vs Current Search (intent to treat) | No AI vs Current Search (LATE) |
|---|---|---|
| Trust in information on Google (7-point scale) | -0.34 points, significant | +0.14, not significant |
| Usefulness (standard deviations) | -0.59, significant | +0.14, not significant |
| Satisfaction (standard deviations) | -0.73, significant | +0.01, not significant |
| Agency (standard deviations) | -0.66, significant | +0.19, not significant |
| Personalized and relevant (standard deviations) | -0.42, significant | +0.23, not significant |
| Intention to switch to Bing (7-point scale) | +1.25 points, significant | +0.10, not significant |
| Preference for AI Mode in news comparisons | -0.12, not significant | -0.16, not significant |
| Trust in AI Mode in news comparisons | -0.08, not significant | -0.15, not significant |
The intention question was specific: "If your search experience from the last 7 days remained permanent, how likely would you be to switch to Bing?" Participants assigned to AI Mode scored 1.25 points higher on a seven-point scale than the control group. The behavior matched the stated intention, at least in part. The share of AI Mode participants who ran any search on Bing, DuckDuckGo or Yahoo during the week rose by 11.2 percentage points.
Does that mean Google would lose users if AI Mode became the default? The paper is careful not to claim it. Its authors write that "If users remain firm in their mistrust and dislike of AI Mode, Google will face a tension between pushing the product on users (to acclimate them as much as for its other benefits), and risking alienation that could cost it ground to competitors in a search market that today is nearly monopolistic." The study measured one week, with users who had not chosen AI Mode and had been told it was mandatory for the study.
The question has some weight because of what Google has said about its plans. In September 2025, Google DeepMind product manager Logan Kilpatrick wrote that AI Mode would become the default search experience, before saying two days later that he had not meant AI Mode was replacing the main search page. On May 19, 2026, Google said AI Mode had passed 1 billion monthly active users globally, with queries roughly three times as long as traditional searches.
What participants said
The 309 participants in the AI Mode group were asked one open question at the end: "What was your overall impression of Google's AI Mode after one week of using it?" The researchers coded each answer with one to four of 17 themes, developed until no new themes emerged.
| Theme in AI Mode participants' answers | Share | Count |
|---|---|---|
| Negative overall sentiment | 33.6% | 103 |
| Positive overall sentiment | 29.3% | 90 |
| Mixed sentiment | 13.7% | 42 |
| Loss of control and agency | 17.6% | 54 |
| Difficulty reaching specific websites | 15.3% | 47 |
| Efficiency and time saving | 14.0% | 43 |
| Limited links or source diversity | 13.4% | 41 |
| Answers too verbose | 6.2% | 19 |
| Faithful summarization | 6.2% | 19 |
| Hallucination and accuracy concerns | 4.6% | 14 |
| Adapted to AI Mode over time | 3.6% | 11 |
| Trust in AI results | 1.6% | 5 |
| Prefer AI summaries to traditional search | 1.6% | 5 |
The quotes the authors chose are blunt. "I like to choose when I use AI," one participant wrote. Another described how "when I just wanted to land on the webpage I would type in, it would tell me about the website instead." On length: "it gives way too much info for easy searches." On accuracy: "the AI mode provided a lot of inaccurate and inconsistent information." The positive side was represented too: "the AI does a strong job of pulling exactly what I was looking for, saving time," and "at first the AI Mode threw me off, but I got used to it quickly."
The navigation complaint is worth noting for anyone who studies branded search. Typing a site's name into Google to reach it is a common habit. Under the AI Mode condition, that habit produced a description of the site rather than a link to click, and 15.3% of respondents raised it without being prompted.
The ad clicks that were not there
For advertisers, the most striking number in the paper is also the least informative. The share of AI Mode participants who clicked any ad fell by 42.7 percentage points. The authors give the reason directly: "The fraction of users clicking on ads decreased by -42.7pp because AI Mode did not surface ads at the time of our experiment." In the discussion they repeat that "at the time of our study, AI Mode did not yet surface ads or rely on other traditional ad-revenue mechanisms," and describe their results as a "pre-monetization baseline."
That description is hard to square with Google's own timeline. Robby Stein, Google's vice president of product for Search, said in an interview recorded on October 30, 2025 that Google had started some experiments on ads within AI Mode. On February 11, 2026, about five weeks before recruitment began, Google introduced a sponsored shopping ad format in AI Mode, with Target and B&H shown as example advertisers. Liz Reid had said in June 2025 that ads would begin rolling out in AI Mode for queries about dining, shopping and purchases, without giving a date.
The paper does not mention any of these. It does not explain how the authors established that AI Mode showed no ads to their participants, and it does not report how many of the AI Mode responses captured by the extension contained a sponsored unit, even though the extension stored the HTML of every response. It is possible that the February format had reached few or none of the participants' accounts by late March, but the paper does not say so.
By the time the paper was posted, Google's ad plans for AI Mode had moved further. At Google Marketing Live on May 20, 2026, the company introduced Conversational Discovery ads and Highlighted Answers for testing in AI Mode in the United States. A few days later, an AI Mode answer to a coffee maker query showed a paid product listing in the middle of the response, marked only by a small "Sponsored" tag. The authors acknowledge that this will change what they measured: "we anticipate user satisfaction and other measures will shift away from this pre-monetization baseline."
For paid search, then, the paper says one narrow thing. In the AI Mode experience participants saw in March 2026, ad clicks carrying a gclid almost disappeared. It says nothing about how ads perform in AI Mode now, and the No AI condition found no significant change in ad clicks when AI Overviews were partly removed.
How the findings sit next to earlier evidence
This is not the first experiment of its kind in 2026, although the paper does not mention its predecessor. Saharsh Agarwal of the Indian School of Business and Ananya Sen of Carnegie Mellon University ran a randomized experiment that hid AI Overviews from some US users between January 7 and February 10, 2026, with 1,065 participants in its analysis. Where an AI Overview would have appeared, outbound organic clicks per search were 0.37 with the Overview and 0.62 without it, a reduction the authors put at 39.8%. Their study, first posted to SSRN on April 3, 2026, also included an exploratory AI Mode arm, which recorded 0.36 external clicks per search against 0.53 in the control group and a drop in satisfaction. By PPC Land's arithmetic, that is about a third fewer clicks per search. The two studies define their click measures differently, so the figures are not directly comparable, but both point the same way on AI Mode.
The two experiments differ on AI Overviews in an instructive way. Agarwal and Sen's hiding intervention ran in January and February. Wang and colleagues' version broke in March after Google changed the Overview code. The second study's weaker AI Overview result may say more about the broken filter than about AI Overviews, and the authors do not claim otherwise.
Wang and colleagues do cite observational work. Pew Research Center found that Google users clicked a traditional result link on 8% of visits that showed an AI summary, against 15% of visits without one, based on 68,879 searches by 900 US adults in March 2025. Google called that study's methodology flawed and its query set unrepresentative. The paper also cites a study estimating that AI Overviews reduced English Wikipedia traffic by 15%.
Publisher data point the same direction, although it cannot isolate the cause. Chartbeat figures published in March 2026 showed search referrals down 60% for small publishers, 47% for medium ones and 22% for large ones over two years. An analysis of 44 large US publishers by XSquareSEO, using Semrush estimates, found aggregate organic visits up 5.02% between two two-year periods, with large gains for some sites and losses of more than half for others. Observational numbers like these can be moved by many things at once. The value of a randomized experiment is that it removes most of those other causes, at the price of a small, unrepresentative sample and a short window.
Why it matters for marketers and publishers
The paper speaks to a commercial argument with legal consequences. Penske Media, owner of Rolling Stone, filed a 101-page antitrust complaint against Google in September 2025 over its use of publishers' content in AI answers. Causal evidence that AI search features reduce outbound clicks bears on the substitution claim at the center of disputes like that one.
The authors also connect their results to the legal status of AI answers in Europe. They write that "a recent German court ruling held that Google's AI Overviews cause it to function as a content provider rather than a neutral intermediary." On May 28, 2026, the Munich Regional Court I issued a preliminary injunction holding Google liable for AI Overview statements about two Munich publishing companies, and its written reasons found that the Overviews were Google's own content, denying it the hosting privilege. The reference the paper actually gives for its sentence, however, is a July 2026 Reuters report headlined "German media regulator says Google's AI Overviews subject to German media law," which concerns a regulator, not a court. The paper does not name a court or a case.
For search marketers, the practical reading is narrower than the headline. The study measured clicks per search, not what happened after the click. Google's claim that AI features produce "higher quality" clicks is about what visitors do once they arrive, and the experiment did not test it. What the experiment does show is that, for a week in March 2026, a forced move to AI Mode reduced the number of times people left Google, and reduced it across news, community and reference sites alike. For publishers whose revenue depends on page views, and for advertisers who buy traffic on those pages, the number of visits is the first thing that matters.
The authors' own conclusion is that the AI search trajectory "presents a clear tradeoff: as Google increasingly answers users' questions directly, referral traffic to the broader web declines substantially, yet these changes do not currently yield commensurate improvements in users' trust or overall search experience." They argue that AI search "should be evaluated not only in terms of the quality of answers generated, but also in terms of how these systems reshape the relationship between users, publishers, and the broader web."
Discrepancies in the paper
- The discussion describes the competitor-search result as "an 11.2% increase in competitor search use," while the results section and Table S4 report it as 11.2 percentage points.
- The German legal example is described as a "court ruling" but cited to a Reuters report about a media regulator.
- The qualitative shares match a base of 307 responses rather than the 309 stated (for example, 103 negative responses is 33.6% of 307 and 33.3% of 309, by PPC Land's calculation).
- The main text gives a baseline average of 4.1 clicks a day for all participants, while the covariate balance table gives group averages of 3.13 to 3.46; the paper does not explain the difference.
- The replication dataset on Harvard Dataverse carries an earlier title: "AI in Search Reduces Traffic to Publishers Without Improving Searcher Experience: A Field Experiment."
- The compliance-adjusted estimation method, the continuous compliance measure and the exclusion of users with no AI Overview searches were not in the preregistration, as the paper itself states.
What the paper does not establish
The study covers seven days of treatment, US participants recruited online, Chrome users who already preferred Google, and a sample younger, more educated and more Democratic than the country. Browser extensions do not run in Chrome's mobile apps, and the paper does not discuss mobile searching. AI Mode was imposed on everyone in that group rather than chosen, from a starting point of 0.6% use. The AI Overview estimate rests on a filter that failed midway and on an estimation method chosen after the fact. The paper measures clicks, not revenue, time on site or conversions, and because the study was not powered to detect small effects on secondary outcomes, the authors warn that "null effects on these outcomes should not be interpreted as evidence of no effect." It has not been peer reviewed.
Key facts
- Title: "AI in Search Reduces Publisher Referrals Without Improving User Experience: Experimental Evidence"
- Identifier: arXiv:2608.18352v1, DOI 10.48550/arXiv.2608.18352
- Source URL: https://arxiv.org/abs/2608.18352
- Submitted: Tuesday, August 18, 2026, 22:08:10 UTC; version 1 is the only version listed as of November 17, 2026
- Categories: Information Retrieval (cs.IR, primary); Computers and Society (cs.CY); Human-Computer Interaction (cs.HC)
- License shown on arXiv: CC BY-NC-ND 4.0
- Status: preprint, not peer reviewed; PDF pages watermarked "Draft"; 46 pages including supplementary materials
- Authors: Stephanie T. Wang (University of Pennsylvania, corresponding author) and Jeffrey Gleason (Northeastern University), equal contribution; Yakov Bart (Northeastern, D'Amore-McKim School of Business); Christo Wilson (Northeastern, Khoury College); Danaé Metaxa (University of Pennsylvania)
- Funding: National Science Foundation award IIS-2442711; NSF Research Traineeship (PEAR) for Gleason; no competing interests declared
- Design: preregistered randomized field experiment on the Open Science Framework; three conditions (No AI Search, Current Search, AI Mode Search); AI Mode enforced through URL parameter udm=50
- Fieldwork: recruitment March 17 to 19, 2026; 3-day baseline plus 7-day treatment
- Samples: 1,444 enrolled; 1,100 behavioral; 956 survey
- Main estimates: AI Mode click-through rate -18.8 points; No AI click-through rate +8.8 points (compliance-adjusted) or +2.7 points (intent to treat, not significant)
- Compliance: 94.7% of AI Mode group searches routed to AI Mode; 51.1% of AI Overviews hidden in No AI group
- Extension: published on the Chrome Web Store under ID cedfhkdnooeicfiaheamppdckbidiaah
- Data: aggregated data and code on Harvard Dataverse; raw browsing data and page snapshots not released for privacy reasons
- Ethics: approved by the Institutional Review Boards of both universities
Timeline
- May 2024 - Google introduces AI Overviews, according to the paper
- March 2025 - Google introduces AI Mode, according to the source the paper cites
- June 25, 2025 - Liz Reid says ads will begin rolling out in AI Mode, without a date
- July 22, 2025 - Pew Research Center finds Google users click result links on 8% of visits with an AI summary and 15% without
- July 2025 - Google disputes Pew's methodology
- August 6, 2025 - Liz Reid says organic click volume is relatively stable and quality clicks are slightly up
- September 6, 2025 - Logan Kilpatrick says AI Mode will become the default, then clarifies two days later
- September 12, 2025 - Penske Media sues Google over AI content practices
- October 30, 2025 - Robby Stein says Google has started experiments with ads in AI Mode
- January 7 to February 10, 2026 - Agarwal and Sen run their AI Overviews experiment with an exploratory AI Mode arm
- February 11, 2026 - Google introduces a sponsored shopping ad format in AI Mode
- March 17, 2026 - Chartbeat data show search referrals down 60% for small publishers over two years
- March 17 to 19, 2026 - Wang and colleagues recruit participants; each then spends three days at baseline and seven days in an assigned condition
- March 2026 - A Google change to AI Overview HTML breaks the study's hiding filter, according to the paper
- April 3, 2026 - Agarwal and Sen post their study finding AI Overviews cut outbound organic clicks 39.8%
- May 19, 2026 - Google says AI Mode has passed 1 billion monthly active users
- May 20, 2026 - Google introduces Conversational Discovery ads and Highlighted Answers for testing in AI Mode
- May 25, 2026 - A paid product listing is seen in the middle of an AI Mode answer
- May 28, 2026 - Munich Regional Court I issues a preliminary injunction against Google over AI Overview statements, finding the Overviews are Google's own content
- July 2026 - Reuters reports that a German media regulator says AI Overviews are subject to German media law, the source the paper cites
- July 17, 2026 - Nick Fox says Google sends billions of clicks a week through AI features
- July 18, 2026 - Google publishes an explainer on the gclid parameter
- August 18, 2026 - Wang, Gleason, Bart, Wilson and Metaxa post the paper to arXiv
- November 17, 2026 - arXiv still lists version 1 as the only version
Related PPC Land coverage
- AI Overviews cut publisher clicks 39.8% in first randomized study - The Agarwal and Sen experiment that hid AI Overviews from US users and measured outbound clicks.
- Researchers find Google AI Overviews cut publisher clicks 39.8% - Detailed coverage of the same study, including its exploratory AI Mode arm.
- AI search summaries reduce link clicks for Google users - Pew Research Center's browsing study of 900 US adults.
- Google disputes Pew study showing AI Overviews reduce clicks by half - Google's response to the Pew findings.
- Google executives contradict research on AI search traffic impact - Liz Reid's August 2025 statement on click volume and click quality.
- Google touts billions of weekly AI Search clicks as CTR data drops 58% - Nick Fox's July 2026 claim set against Ahrefs click-through data.
- Google AI Mode hits 1 billion users - how search changed in one year - Google's first-year usage report for AI Mode.
- Google unveils shopping ads in AI Mode, doubling down on conversational commerce - The February 2026 sponsored shopping format in AI Mode.
- Google's new AI search ad formats reshape how brands reach buyers - The AI Mode ad formats introduced at Google Marketing Live 2026.
- Small publishers lost 60% of search traffic as AI reshapes the web - Chartbeat referral data by publisher size.
- Munich court holds Google liable for AI Overviews defamation - a first - The German ruling on false statements in AI Overviews.
- Penske Media files antitrust lawsuit against Google over AI content practices - The publisher lawsuit over Google's use of content in AI answers.
Summary
Who: Stephanie T. Wang and Danaé Metaxa of the University of Pennsylvania, and Jeffrey Gleason, Yakov Bart and Christo Wilson of Northeastern University, with 1,100 US Google users recruited mainly through Prolific.
What: A preregistered field experiment that used a browser extension to switch participants' Google searches entirely to AI Mode, remove AI Overviews, or leave Google unchanged. AI Mode cut click-through to external sites by 18.8 percentage points, reduced visits to news sites, Reddit and Wikipedia, lowered trust and satisfaction, and raised use of rival search engines by 11.2 points. Removing AI Overviews raised click-through by an estimated 8.8 points after adjustment for a filter that failed mid-study; the unadjusted effect was not significant.
When: Participants were recruited March 17 to 19, 2026, and each took part for ten days. The paper was submitted to arXiv on August 18, 2026 and has not been revised or peer reviewed.
Where: Google Search in the United States, on participants' own Chromium browsers; the paper was posted to arXiv by researchers in Pennsylvania and Massachusetts.
Why: Publishers and Google disagree on whether AI answers in search replace visits to websites or lead to them. The study offers causal evidence on that question for AI Mode, weaker evidence for AI Overviews, and a snapshot of AI Mode taken, according to its authors, before it carried ads.
Discussion