Google Joins OpenAI in Backing New robots.txt AI Training Rule

Google has announced that it is joining OpenAI, Anthropic and Microsoft in supporting a new directive for the robots.txt file. It is designed to give website owners more control over whether their content is used to train generative AI models.

Until now, publishers had to write separate rules for each company's crawler if they wanted to keep their pages out of AI training. The proposed system replaces that with a single, unified signal that participating companies can recognize and respect.

How it works

Site owners will be able to use a dedicated user agent to mark pages or sections they do not want used for AI training. They can still allow access to specific parts of a site, such as news pages or material meant for publication.

The companies stressed that the directive does not stop sites from appearing in search results. It targets only data collection aimed at training AI models. For Google, the new signal will be treated as an instruction for its Google-Extended user agent, which gathers material related to training its Gemini models.

Gradual rollout through 2027

Anthropic already supports the directive. Microsoft plans to begin applying it on February 1, 2027, and Google is expected to follow in early 2027. OpenAI says it will implement the system in the coming months.

The directive will not affect data the companies have already collected. Its role is limited to governing future scraping of content.

The move marks a notable shift in the relationship between AI developers and publishers, as demands grow for content owners to have more say over how their work is used to build generative models.