Google and Bing back one robots.txt rule to opt out of AI training

For three years, publishers who wanted to keep their work out of AI training data have had to learn a new crawler name every few months. Google-Extended, GPTBot, ClaudeBot, Applebot-Extended, CCBot: each company built its own switch, and each switch had its own rules about what it covered. On Friday, October 30, the two largest search engines in the Western world said they would honour a single one.

Google said it will voluntarily adopt a proposal under consideration at the Internet Engineering Task Force that lets sites set AI training preferences across several AI providers in a single robots.txt step, Barry Schwartz of Search Engine Roundtable reported. The mechanism is a shared policy token, written in robots.txt as a user-agent called "__usage_ai-training". A site that disallows that token tells every participating company that its content should not be used to train models. "This will help websites express their AI training preferences across multiple AI providers in a single step," Google said in its announcement on the Search Central blog.

The timetable is not immediate. "We aim to support this new control along with other AI providers in 2027, and will provide updates on our progress," Google wrote. Its documentation for common crawlers now says early 2027. John Mueller added a detail that matters for anyone who has already configured Google-Extended: if a site uses the new token to opt out and does not explicitly mention Google-Extended, Google will treat the site as if Google-Extended were disallowed. Existing company-specific controls will keep working, and the opt-out in Search Console remains.

The proposal itself is a draft at the IETF, produced by the AI Preferences working group, known as AIPREF. OpenAI has published its own commitment to the shared signal, and Search Engine Roundtable reported that Anthropic is also participating. Nate Hake, a publisher who has taken part in the working group, cautioned on X that the work is unfinished and some disputes remain unresolved. The difference between a draft and a standard is large. A draft can change wording, scope and syntax before it is adopted, and the companies have committed only to supporting the signal, not to a fixed definition of what training means.

Microsoft went further on the same day. Krishna Madhavan, principal product manager at Bing, announced new Bing AI content controls that separate training from grounding, while keeping eligibility for ordinary search results. Grounding is the process by which an AI assistant retrieves pages at the moment it answers a question, then cites or summarises them. Training is the earlier process of building the model. Until now, most publishers have had to treat them as one decision.

Bing now offers three settings. AI Training allows or prevents use of content to train models. AI Grounding allows or prevents use of content to help generate AI answers. A third option, Limited Grounding, lets a site support Microsoft's own AI products while refusing Microsoft's third-party grounding services, which other companies buy to give their chatbots access to the web. "Microsoft is joining industry peers in announcing a shared AI training preference in robots.txt," the company said, and it is adding page-level training preferences as well. On search rankings, the message was flat: "The traditional Bing rankings are unaffected."

The Bing schedule is precise. From December 15, 2026, temporary grounding controls will appear in Bing Webmaster Tools for sites that use the NoArchive or NoCache directives. From February 1, 2027, Microsoft will begin honouring the new preferences when it crawls. Publishers relying on the old directives have until March 1, 2027 to configure the new ones, and the temporary grounding preferences expire on September 30, 2027. The example Search Engine Roundtable offered was a publisher that blocks training but still lets Copilot cite its articles. The trade-off is explicit: blocking AI answers can reduce visibility in AI experiences.

What the two announcements share is a recognition that the old binary of crawl or do not crawl no longer describes the choices publishers face. What they do not yet share is a common vocabulary beyond training. Google's token covers training alone, while Bing's controls distinguish training from answers. A site that wants to appear in AI Mode citations while staying out of Gemini's training data still has to read two sets of documentation.

A study puts AI Overviews click-through at 0.9%

The value of appearing in AI answers is the question behind every one of those settings, and new evidence arrived the same morning. Cyrus Shepard published a small case study on LinkedIn in which he used Google's opt-out controls to remove a site from AI Overviews and AI Mode, then compared the traffic before and after. Search Engine Roundtable reported that AI Overviews and AI Mode produced an estimated click-through rate of about 0.9%. "AI impressions had an estimated ~0.9% CTR based on analyzing overall before/after click data (not too shabby)," Shepard wrote.

The finding that drew attention was the second one. When the site was removed from the AI features, its AI impressions fell immediately to zero, as expected. But ordinary organic clicks fell too. Regular organic clicks "ALSO DROPPED when AI features were disabled," Shepard wrote. AI impressions had made up about 8% of the site's total impressions. The URLs with the most AI visibility lost ground and then recovered when visibility was restored. Shepard called the pattern a "Search SERP Halo Effect". Greg Sterling of Near Media suggested in the comments that a presence in AI Overviews seems to create brand lift that leads to later searches and clicks.

Schwartz listed the limits. The sample is one site. AI Overviews still appeared on the results pages during the test; only this site was absent from them. And sitewide average position rose during the period because of new posts and brand searches, which complicates any before-and-after comparison. Still, for publishers weighing the new Google and Bing controls, the study offers a rare measurement of what opting out costs, and it suggests the cost is not limited to the AI features themselves.

Bing tests ads inside its AI Overviews

Microsoft appears to be testing the commercial side of the same feature. Khushal Bherwani posted a screenshot on X showing a block labelled Sponsored Results inside a Bing AI Overview, and Search Engine Roundtable reported that Bing is showing ads within its Copilot-powered AI Overviews. "Bing is showing Sponsored Results within AI Overviews," Bherwani wrote. Schwartz said he had not seen ads within AI responses on Bing himself and could not trigger them. Microsoft has made no statement.

Bing renamed its Copilot Search box to AI Overviews a few months ago, borrowing Google's term. Placing sponsored results inside the generated summary would bring it closer to Google's model as well, where ads now appear above, below and within AI Overviews in many markets. For Microsoft Advertising, the generated summary is where more queries end, so it is also where the inventory has to be.

Gemini drops its UTM tags

Measuring visits from AI assistants became harder again. The SEO Natzir found that Gemini stopped attaching UTM parameters to cited links around September 28. Google had begun appending "utm_source=gemini" to citations in early September, a change Search Engine Journal reported at the time. Natzir's chart showed tagged visits growing for three weeks and then vanishing overnight. "Google seems to have stopped appending UTM parameters on Gemini links," Schwartz wrote, adding that he no longer sees them on any Gemini link.

The loss is partial. Visits with a gemini.google.com referrer on the web stayed flat, because the desktop version still passes referrer data. The bigger gap is on phones. Natzir found that Gemini's iOS app opens links in its own embedded browser with no referrer, so those visits land in Google Analytics 4 as Direct traffic. The only trace is a GeminiiOS user-agent in server logs. For marketers trying to size AI referrals, that means some Gemini traffic is invisible to standard analytics and counted as visitors who typed the address themselves.

Google warns that AI made bot impersonation worse

Gary Illyes of Google said in an interview with Aneta Samkoff on the Odys Podcast that AI has made bot impersonation worse. Rogue crawlers that claim to be Googlebot have multiplied as companies harvest content for training. "I think it is AI," Illyes said when asked about the cause. Firewalls and content delivery networks catch much of the traffic, but not all. "Anyone can impersonate Googlebot in about 30 seconds," he said, by changing a user-agent string. His own honeypots attracted requests claiming to be Googlebot from IP addresses unrelated to Google.

His advice was practical. Site owners should verify Googlebot by IP address and reverse DNS, not by user-agent. Robots.txt is a suggestion, not a security control, and real protection comes from a web application firewall or a content delivery network. Illyes discussed Web Bot Auth, a scheme for cryptographic crawler signatures, and said he is exploring an early-stage crawler reputation system, while warning that a system controlled by one company could introduce bias. For most small sites, he said, the first step is to check whether bot traffic is actually causing a problem.

The interview sits uneasily beside the day's announcements. The new robots.txt token and Bing's controls depend on crawlers that identify themselves honestly and obey the file. Illyes was describing the crawlers that do neither.

Zocdoc opens its doors to AI agents

Outside search, another marketplace prepared for software customers. Zocdoc, the app that connects patients with healthcare providers, said it has opened its platform to AI agents, Katie Duffy of Adweek reported. Agents can connect to Zocdoc's systems to find providers, verify insurance and book appointments, then add them to a patient's calendar. Adweek's example was a request typed into an agent: "I need a dermatologist near 10013 who takes Aetna, ideally this week." The agent searches Zocdoc's database for a match on location, specialty and insurance, then makes the booking.

Zocdoc has built a "Welcome, agents" page that lets visitors switch from a human mode to an agent mode, and that generates text people can paste into their own assistant. The move turns the marketplace into a back end for any agent, not only for the search and chat products it has partnered with before. For healthcare marketers, it raises a familiar question in a new setting: when an agent chooses the provider, the ranking rules inside the agent matter more than the listing.

Agencies adopt AI agents but keep their org charts

Inside agencies, the adoption of AI has outrun the reorganisation that might follow it. Sam Bradley of Digiday reported from a townhall session at Digiday's Media Buying Summit in Newport Beach, California, that agencies have embraced AI agents but left team structures, job descriptions and career paths largely unchanged. Attendees spoke under Chatham House Rules, and some described colleagues using AI as an all-purpose corner cutter that skips critical thinking.

Gila Wilensky, executive vice president and head of data and media for North America at Monks, said training junior staff and retraining senior colleagues is part of the answer. "There is a baseline that everyone is expected to understand and be fluent in," she said. "If we don't provide that, things can go off the rails." Nadja Bellan-White, group chief executive of M&C Saatchi North America, said her agency has set aside an hour a week on employee timesheets for training. Wilensky predicted that checking AI output will become a career in its own right.

The article cited a May Digiday survey in which only 9% of media agencies said they were not pursuing agentic AI, and an Apollo estimate that profit margins and revenue per worker have not improved outside the technology sector since 2022. Some firms give developers and engineers access to more expensive models while restricting others, mainly to control compute costs. The gap between enthusiasm and productivity is the tension in the headline.

Streaming audiences that exist only on paper

Connected television buyers were warned about the difference between a large audience and an available one. In an AdExchanger column, Justin Fromm, head of insights and thought leadership at Samsung Ads, argued that ghost audiences are haunting CTV measurement. He defined them as subscribers who remain on a service's books but do not watch during the three or four months of a typical campaign. "Just because a platform's audience is large doesn't mean it's available," Fromm wrote.

The scale of ad-supported streaming makes the point pressing. Fromm cited Antenna data showing that 48% of US premium subscription video accounts were ad-supported in the first quarter of 2026, that 59% of new subscriptions that quarter were for ad tiers, and that ad-supported premium subscriptions reached 110 million, up from 53 million two years earlier. CTV upfront spending surpassed linear television this year, and Omdia projects global CTV ad revenue will rise from $44 billion in 2025 to $81 billion by 2030. Samsung Ads found that two of every three new viewers who streamed NFL games had stopped using the service after the season, nearly twice the churn of existing users. "Viewer loyalty is to the show, not the app," Fromm wrote. He proposed that buyers judge platforms by availability and addressability rather than reported size.

Brands chase fans at the marathon, the racetrack and Bravo

Sponsorship is moving towards fandom, and three stories on Friday showed different routes in. The New York City Marathon celebrates the 50th anniversary of its five-borough course on Sunday, November 1, and Modern Retail's Gabriela Barkho, writing in Digiday, reported that more brands are lining up around it, from footwear to beauty and restaurants. Maybelline is the race's first official beauty sponsor. The energy drink Celsius opened Club Celsius in Brooklyn, running from October 30 to November 2. "The brand was founded in fitness, and it remains an integral part of our DNA," said Kyle Watson, its chief brand officer. The mattress brand Saatva and the apparel brand Vuori are holding their first marathon activations.

SharkNinja took a longer view. Digiday's Future of Marketing Briefing, by Krystal Scanlon and Seb Joseph, examined its five-year sponsorship of McLaren, covering four racing series, McLaren's engineers and direct access to the drivers Lando Norris and Oscar Piastri. SharkNinja signed in mid-September, with most of the Formula 1 season already run. Laura Dyer, its senior vice president of global brand and partnerships, said the aim is younger fans, and that she was wary of chasing a single cultural moment. "The movie was great but it spiked and then it left," she said, apparently referring to the brief lift from the F1 film. Engineers from both companies will meet for a set number of hours each year, with battery life an early focus. Dyer described the engagement yardstick as "really wooly."

Bravo built a festival for its own fans. Bianca Brutus of Adweek reported that Bravo Fan Fest returned in Charleston, South Carolina, on October 24 with five sponsors: Colgate Optic White, Planet Fitness, State Farm, Wayfair and Wendy's. Wendy's served two new Frosty flavours inspired by Real Housewives cast members from a food truck, Planet Fitness handed out bottles with a hang tag that unlocked a free month of its Black Card, and Wayfair's gift boxes carried a shoppable QR code. "Each partnership brings the brand into the Bravo world through familiar Easter eggs and fan-favorite moments," said Sari Feinberg, senior vice president of marketing and brand partnerships at NBCUniversal.

The Cheesecake Factory wins a political food fight by saying nothing

A restaurant chain became the subject of a presidential exchange and kept quiet. In an Adweek opinion piece, Nataly Kelly, chief marketing officer of Zappi, argued that The Cheesecake Factory is the real winner in the Obama-Trump food fight. On Saturday, October 24, Barack Obama told a crowd in Cleveland that the White House looked like "Saddam Hussein opened a Cheesecake Factory." By Monday night, Donald Trump had called the chain "a very successful company" at a rally in Mississippi, and Treasury Secretary Scott Bessent ate at one that day.

Google searches for the brand rose 70% week on week and 130% on the same week a year earlier, Kelly wrote. The company did not comment, and no one mentioned the episode on its Tuesday earnings call, where revenue passed $1 billion for a second consecutive record quarter and comparable sales rose 15.2%. David Overton, the chief executive, said only that the brand "has always had broad appeal." On Wednesday, Obama told a rally in Georgia that the chain had been his "jam for about a decade." Kelly's argument is that silence works only when brand equity is strong enough to absorb attention from both sides.

Patrick Stewart's work in advertising

Patrick Stewart died at the age of 89, Adweek reported on October 30. Robert Klara's tribute traced the brand work Stewart lent his voice and presence to, from Pontiac to Paramount. The most recent was Paramount's 2024 Super Bowl commercial, in which Stewart shrugged off a shearling coat to reveal 1920s football gear and threw a pass alongside Drew Barrymore, Jeff Probst, Peppa Pig and characters from the studio's catalogue. The spot used his gravity as the joke, a role he played in advertising for decades.

Campaigns, hires and holiday tricks

Adweek's weekly selection of campaigns that caught its eye included the singer Lisa in a PlayStation spot, Owen Thiele, Tove Lo and Ken Eurich for Tinder, an e.l.f. haircare ad featuring an astronaut, Gusto's salute to small businesses and a Take Back the Night campaign that "calls attention to the scandal of sexual assault on American campuses." The most effective ad of the week, measured with EDO, featured the M&M's Spokescandies.

Rebecca Stewart's roundup of recent marketer moves covered three weeks of leadership changes, including Coursera's appointment of Amy Weisenbach as chief marketing officer and changes at Beast Industries and The New York Times.

And for Halloween, both search engines added a seasonal touch. Search Engine Roundtable reported that Google and Bing launched Halloween easter eggs. Google's invites searchers to find a costume using AI Mode, which Schwartz described as boring. Bing's shows a floating ghost that bursts into candy when tapped. Even the decorations now point towards the AI product.

What comes next

The calendar ahead is set mostly by the AI controls. The New York City Marathon runs on November 1, and the midterm elections follow on November 3. Bing's temporary grounding controls arrive in Webmaster Tools on December 15. Microsoft begins honouring the new preferences on February 1, 2027, with a March 1 deadline for sites on the old directives, and Google aims to support the shared training token in early 2027. Between now and then, the IETF draft can still change. Publishers will be deciding how much of their work to offer to AI systems based on evidence such as a single-site study of 0.9% click-through, while the systems themselves keep removing the tags that would let them measure the traffic.