News & analysis
Google, Microsoft, OpenAI Back Shared AI Training Opt-Out
Google, Microsoft, and OpenAI said Friday they plan to honor a single robots.txt line that lets a site state one AI training preference instead of writing separate rules for each crawler.
By Roomer Editorial · · 4 min read
What did Google, Microsoft, and OpenAI announce?
On Friday, the three companies said they plan to honor a shared robots.txt line covering AI training preferences. A site could set it once and have every participating provider read the same instruction, rather than maintaining separate blocks for each company's crawler.
The line comes from an Internet Draft submitted to the IETF's AI Preferences working group by Anthropic's Kevin Kelley. The IETF's document tracker lists it as an individual draft, which is not official standardization. Kelley told the working group's mailing list that Anthropic has started building support for it.
None of the three companies currently supports the line, based on the timelines each gave.
How does the shared opt-out line actually work?
The draft doesn't add a new robots.txt directive. It defines a policy token, `__usage_ai-training`, that a site writes as an ordinary rule group using the existing Allow and Disallow syntax. The draft's example opts a site out of training everywhere except a `/press-kit/` folder:
``` User-Agent: __usage_ai-training Disallow: / Allow: /press-kit/ ```
The token's meaning comes from the working group's vocabulary draft, which defines AI training as using content to change a generative model's learned parameters. Non-generative uses, like classification or ranking, are excluded.
Existing entries in a site's robots.txt file don't factor into this preference. The draft specifies that a blanket `User-Agent: *` rule never applies to the token, so leaving the `__usage_ai-training` group out entirely means no preference has been stated — Kelley notes that's the current status of every published robots.txt file, since the token didn't exist before now. None of this changes how crawling works.
Where do Google and Microsoft diverge?
Google plans to treat a site's shared opt-out, when Google-Extended isn't separately specified, the same as a Google-Extended block. That control also covers grounding inside Gemini Apps — Google's term for supplying search-index content to the model when it answers a query. Google aims to support this in early 2027. Google-Extended has no effect on inclusion in Google Search; AI Overviews and AI Mode use a separate opt-out in Search Console.
Microsoft is folding NoArchive and NoCache into the new system and wants sites using those tags to switch by March 1, 2027. It's adding a distinct AI Grounding preference, set at the page level, governing whether content can inform AI-generated answers. A new `DisallowExtended` setting blocks grounding through third-party services built on Microsoft's grounding infrastructure while still allowing it in Microsoft's own tools like Copilot. Microsoft starts honoring the new settings February 1, 2027, and stops reading the old tags after March 1, 2027. A temporary Bing Webmaster Tools setting opens December 15 for sites still relying on the old tags.
OpenAI says it aims to honor the signal within the next few months, while its current robots.txt policy stays in effect meanwhile. It hasn't said what happens if the shared line conflicts with a site's existing GPTBot rule. As of October 30, Anthropic's newsroom had no public post about the signal, though Kelley's message to the working group's mailing list said rolling out support across Anthropic's services will take time. Anthropic currently offers only three crawler controls: ClaudeBot, Claude-User, and Claude-SearchBot.
Does the new line replace what sites already do?
Not evenly. Microsoft is the only company replacing controls sites currently use — NoArchive and NoCache stop working for this purpose after March 2027. Google, OpenAI, and Kelley all say existing controls keep functioning alongside the new line.
The line also doesn't map cleanly onto each company's actual uses of content. Microsoft treats training and grounding as separate settings. Google bundles both under Google-Extended, with no documented way to block Gemini training while still allowing grounding in Gemini Apps.
Roomer Editorial's take
A travel creator's own site is exactly the kind of property this affects. Roomer builds creators a site from their videos, with bookable pages drawing on inventory from Vio, Stay22, GetYourGuide, Viator, and Klook, and those pages get indexed — Roomer has roughly a million pages indexed on Google already. That means the Google-Extended question isn't abstract for this audience.
Roomer Editorial's view: don't rush to flip a blanket AI training opt-out once this ships. Because Google ties training and grounding together, opting out could also pull your bookable hotel page out of the pool Gemini Apps draws on when answering a traveler's question — the same kind of visibility you're trying to build by having a site in the first place. Microsoft's split setup gives more room to block training without touching grounding, which is the more defensible default for a creator who wants their reviews quoted but doesn't want them training a model.
The action: once these opt-outs actually ship in 2027, check whether your site uses Google-Extended, NoArchive, or NoCache today, and decide training and grounding as two separate questions rather than one switch. Don't set it now based on today's announcement — nothing changes until each company turns support on.
Source: https://www.searchenginejournal.com/google-microsoft-openai-shared-ai-training-opt-out/595387/
