OpenAI, Google, Microsoft and Anthropic back shared AI opt-out signal
OpenAI, Google, Microsoft and Anthropic say they will adopt a shared signal in robots.txt that lets websites state their AI training preferences. Anthropic has already adopted it, and the others plan to follow.
- Four AI companies say they will adopt a shared robots.txt signal for AI training preferences.
- Anthropic has adopted it; Google and Microsoft plan to support it next year.
- OpenAI says it will implement it in the coming months.
OpenAI, Google, Microsoft and Anthropic have said they will adopt a new standard that lets publishers opt out of having their sites scraped for AI, The Verge reported. OpenAI said it will honor a shared robots.txt signal that helps website operators communicate their AI training preferences to participating providers.
At present, sites must signal their AI training preferences separately to each AI company. The new control lets publishers apply their preferences across several companies at once instead of setting them one by one.
The shared signal, described as a policy token, was proposed by Kevin Kelley, a technical lead for standards at Anthropic, as part of the Internet Engineering Task Force (IETF). It will live within a website's robots.txt file.
According to The Verge, the four companies say they will restrict their AI crawlers from accessing a site if they encounter the shared token. Engadget also reported that the companies pledged to support the common signal for blocking scraping for AI training.
On timing, Anthropic has already adopted the signal. Google and Microsoft plan to support it next year, and OpenAI says it will implement it in the coming months.
Microsoft will offer more granular controls that let publishers opt out of AI training or AI grounding. The company says websites will still appear in Bing's search results regardless of their preferences. Google had earlier introduced controls letting sites decide whether their content appears in AI search features or is used for AI training, without affecting search visibility.
Meta has not committed to the setting. Its communications head, Andy Stone, signaled support but did not say whether Meta will honor it. He said: "We believe creating a common signal that conveys such preferences to multiple providers at once is an important step". He added: "And we look forward to working with other AI companies on the standards and their implementation across the industry."
The move comes as publishers face a continuing influx of AI crawlers. The Verge noted other tools in this area: the Really Simple Licensing (RSL) standard lets publishers charge crawlers that visit their sites, and Cloudflare blocks AI crawlers by default.
Why it matters
Pakistani websites and publishers can use the signal to state whether their content may be used to train AI models, with one setting rather than separate ones for each company.
