Google to Honor Universal IETF AI Training Token in robots.txt
From 2027, Google plans to read a vendor-neutral IETF token in robots.txt that signals whether content may be used for AI training. Google-Extended stays for exceptions.
Google plans to start evaluating a vendor-neutral AI training token in robots.txt from 2027. The token comes from the IETF, the internet standards body, and is meant to let website operators opt out of AI training with a single entry that applies to all AI developers rather than to one company.
Google-Extended, the company's existing control for AI use of web content, will not disappear. According to the announcement, it remains available for exceptions, so site owners can still set rules specifically for Google where they want to.
Why it matters
Until now, publishers who wanted to keep their content out of AI training had to maintain separate robots.txt rules for each crawler or AI vendor. Every new provider meant another line to add and another name to track. A shared, standardized token would reduce that effort and make the signal easier to understand and to check.
The move also gives the IETF approach weight in practice: a standard only helps site operators if the large crawler operators actually honor it, and Google's commitment is a notable step in that direction.
What comes next
Google names 2027 as the starting point, so website operators have time to prepare. Until then, existing rules, including those for Google-Extended, continue to apply. Whether and when other AI companies adopt the IETF token has not been stated in the available information.
Apps worth a look
SUMO: the number puzzleTap the right tiles, add them up, hit the target. Easy to learn, hard to master. Free, for all ages.
Dialo: learn languages by speakingTalk to an AI conversation partner; every sentence comes back corrected and explained in your language. 9 languages, A1 to C2.
ediviewer: read EDI files like documentsOpen EDIFACT and X12 messages, see every segment explained, convert to JSON/XML, export PDF and Excel. Runs locally, no upload.
More from this day
See all →
Anthropic Shows Unreleased AI Model Exploiting Linux and Chrome Flaws
Shadow AI: Deleting a ChatGPT Chat Doesn't Erase Company Data
JR East Puts Chainsaw-Wielding Humanoid Robot to Work on Rail Lines
