On October 30, OpenAI, Google, Microsoft and Anthropic said they will respect a single instruction that website owners can place in robots.txt, the plain text file sites already use to tell automated visitors which pages to skip. A site that adds a group headed “User-Agent: __usage_ai-training” followed by “Disallow: /” is asking all four companies not to use its pages to train their AI models, and it can leave chosen sections open, such as a press area. The signal comes from a draft written by Kevin Kelley, a standards lead at Anthropic, and submitted to the Internet Engineering Task Force, the body that sets many of the internet’s technical rules. For now the draft has no formal standing there.

The companies have set different dates. Anthropic says it has started adding support, OpenAI promises it in the coming months, Microsoft will apply it from February 1, 2027, and Google says it aims to do so in 2027. Compliance is voluntary, other AI companies are not bound, and Meta has not committed to it. The rule also works only going forward, so pages already collected stay in existing training data. The common rule that blocks every bot at once does not count as a training preference, which means newsrooms, artists and small businesses that want out must add the new entry by name. Microsoft is also adding a second setting that lets a page refuse to feed the answers written by Bing and Copilot.