Google, OpenAI, Microsoft and Anthropic agree on one robots.txt line to say "don't train on this"

Google, OpenAI, Microsoft and Anthropic said on Friday, Oct. 30, 2026, that they will honor a shared robots.txt signal that lets website owners tell all participating AI providers, in one step, not to use their content for generative AI training. It's based on an Internet-Draft from the IETF's AI Preferences working group, published the same day.

For three years, the polite way to tell AI labs "don't train on my site" was to memorize a zoo of crawler names, GPTBot and Google-Extended and ClaudeBot and friends, and hope you didn't miss the new one. One line that every big lab says it will respect is a genuine quality-of-life upgrade for publishers. It's also notable who wrote it. Anthropic's engineer authored the draft and Google, OpenAI and Microsoft signed on the same day, which is about as close to a peace treaty as this fight gets. The fine print matters, though. It's voluntary, it's a draft, most of it doesn't switch on until 2027, and it only covers future training, not the models already built. The labs get to look cooperative while regulators in Europe circle. Publishers get a cleaner off switch. Whether anyone outside the big four honors it is the real test.

The tool is an ordinary robots.txt file, the plain text file a site already publishes to tell automated visitors which pages to skip. The new piece is a shared name in the User-Agent line, the line that usually names one company's crawler. The name is a policy token, __usage_ai-training. The draft's example is a group that reads User-Agent: __usage_ai-training, then Disallow: / , then Allow: /press-kit/. Disallow means skip this. Allow means this part is fine. That example says no AI training on the whole site, except the press-kit folder. It adds no new commands and no new grammar, so a parser that does not know the token simply ignores the group. Those rules are the draft's.

The draft says a wildcard group, User-Agent: *, the line that means every other crawler, must not be read as matching the token. A site's existing rules do not silently become a training preference. If there is no __usage_ai-training group, no preference has been expressed. The token is not a crawling instruction. A rule written for one company's crawler should take precedence over the general one, and a private agreement may override it. Those limits are the draft's.

"AI training," in the definition the draft uses from the IETF's AI Preferences work, means using an asset to change the learned parameters of a generative AI model. Parameters are the numbers inside a model that training adjusts. Generative, here, means a model that produces new writing, pictures, or other content. Training a model used only for non-generative tasks, such as classification, ranking, or scoring, is not covered. That definition is the draft's.

The document is an individual Internet-Draft, the version numbered -00, by Kevin Kelley of Anthropic, dated Oct. 30, 2026. It expires May 3, 2027. An Internet-Draft is a working document posted for discussion. This one is not an IETF standard. Its acknowledgments list contributors from Google, Microsoft, OpenAI, Meta, Cloudflare, News Corp, the BBC, the Financial Times, the Internet Archive, Common Crawl, the Electronic Frontier Foundation, and others. Those lines are the draft's.

Google says that if a site opts out with the shared token and does not explicitly set Google-Extended, Google plans to treat that as if Google-Extended were disallowed. Google-Extended is the robots.txt name Google already uses for training. Controls written for one company keep working. A site that specifically allows Google-Extended stays eligible to appear in grounding responses in the Gemini apps. Grounding, here, means the model can use the page when it answers and point back to it. Google says it aims to support the control along with other AI providers in 2027, and calls the approach consistent with the European Union's General-Purpose AI Code of Practice, which treats robots.txt as a primary way to state a machine-readable preference. Those lines are John Mueller's, on the Google Search Central blog the same day.

OpenAI says it will honor the common signal for foundational generative AI training, and that it aims to implement that "in the coming months." Its existing crawler controls will keep applying. It encourages other AI providers to honor the signal. Those lines are OpenAI's, the same day.

Microsoft, as Search Engine Land reports, will begin honoring the new preferences as it crawls, starting Feb. 1, 2027. It is also adding a page-level switch for AI training, and a separate page-level preference for AI grounding. That grounding preference can be Allow, which is the default, Disallow, or DisallowExtended. DisallowExtended allows Microsoft's own products, such as Copilot, and does not allow grounding services Microsoft provides to other companies. Sites that still rely on the older NoArchive or NoCache directives have until March 1, 2027 to set the new preferences. Those dates and switches are Search Engine Land's account of Microsoft's announcement.

The picture is Figure 1 of the Oct. 30 draft, the robots.txt group with User-Agent: __usage_ai-training, Disallow: /, and Allow: /press-kit/. The credit is the IETF draft, a screenshot.

In plain terms, Google, OpenAI, Microsoft, and Anthropic said on Friday they will honor one shared robots.txt line for generative AI training. The line is defined in a draft by an Anthropic engineer, posted the same day, and it is not a finished standard. Google and Microsoft put most of the switch in 2027. OpenAI says the coming months. The posts describe a preference for training that has not happened yet.