OpenAI, Google and Microsoft back one robots.txt signal for AI training

OpenAI, Google and Microsoft announced on October 30 that they will support a common way for website operators to tell several AI companies whether their content may be used for AI training. Anthropic has also started implementing it.

The mechanism comes from an Internet-Draft titled "Machine-Readable Training Preference Signal", which Anthropic's standards contact Kevin Kelly submitted to the IETF the same day. It builds on discussions in the IETF's AI Preferences (AIPREF) working group. Until now, site owners had to identify each AI company's crawler and write separate rules for it.

How the signal works

Under the draft, operators add a policy token, "__usage_ai-training", to the User-Agent line in robots.txt. "Disallow" under that token means training is refused, while "Allow" means it is permitted. Disallowing "/" and allowing "/press-kit/", for example, rejects training use for everything except the press kit.

The signal is independent of crawl control and does not block crawling itself; site owners who want that must keep using crawler-specific rules. Sites that set no token are treated as having expressed no preference. The draft adds no new directives or syntax, so existing parsers are unaffected. It covers training that changes the parameters of generative AI models. Companies may still give priority to their own controls or honor separate contracts. The draft was submitted by an individual and has no formal IETF standing yet.

Company timelines

OpenAI aims to support the signal within months and will publish details at launch. Google is targeting 2027. If a site rejects training through the shared signal and has not explicitly set Google-Extended, Google will treat Google-Extended as rejected too. Sites that explicitly allow Google-Extended may be used for grounding in the Gemini app and could receive clicks.

Microsoft will begin reading the new settings on February 1, 2027. It will also add page-level controls through meta tags and HTTP headers, which override robots.txt, plus grounding controls including "Disallow" and "DisallowExtended". These replace "NoArchive" and "NoCache", and Microsoft asks sites to migrate by March 1, 2027. Bing search listings are unaffected.

Anthropic says it has started implementing the draft, though extending support across all its services will take time. The move comes amid a wave of publisher lawsuits over AI use of web content.