Microsoft and Google Back Shared Opt-Out for AI Training

Microsoft announced on 30 October new rules that let website owners control how their content is used in the company's AI products. Publishers can now block model training and the use of their pages in Bing and Copilot answers as two separate choices, while staying in regular search results. Google said the same day that it intends to support a common mechanism for opting out of AI training.

Both companies are aligning with a standard being developed at the IETF. It would let site owners set limits for several AI providers at once, instead of writing separate rules for each crawler.

Two separate controls

Microsoft distinguishes AI Training from AI Grounding. Training means text, images and other material are used to teach models, which can later reproduce what they learned without visiting the source. Grounding means the AI consults a page to answer a specific question, for example by summarising an article, using its facts and linking to the source.

Training can be blocked for a whole site with two lines in robots.txt. This does not affect indexing, so the site still appears in Bing unless crawling is blocked. Grounding cannot be set in robots.txt. Owners can use a meta tag in the page's HTML or an HTTP response header, which can be returned on every page to cover a whole site. An intermediate option, ai-grounding:disallowextended, allows Microsoft's own products such as Copilot but bars passing content to third-party developers through services like Web IQ and Foundry API.

The two controls are independent. If an owner specifies nothing, Microsoft treats both uses as allowed.

Timeline and limits

The rules take effect on 1 February 2027 as sites are re-crawled. Owners using the older NoArchive and NoCache settings have a transition period until 1 March 2027 and must replace them. Keeping a grounding ban until re-indexing may require a temporary setting in Bing Webmaster Tools.

Google plans to add support during 2027. If a site blocks training with the new directive and has no separate Google-Extended rules, Google will treat training as blocked; explicit Google-Extended settings will take priority. Google has not announced support for Microsoft's grounding control.

The IETF project is not yet an approved standard, and the directives express a preference rather than technical protection, so compliance depends on the companies. Microsoft warns that blocking grounding will reduce a site's presence in AI answers and potential traffic. The rules provide no compensation for permitted use.