Google, OpenAI and Anthropic Adopt Unified Robots.txt Standard for AI Training

OPA WIRE, Oct 31 (OPA) — Google, OpenAI, Anthropic and Microsoft have agreed to support a new robots.txt directive designed to streamline how publishers manage AI data harvesting. The initiative adopts a proposal from the Internet Engineering Task Force (IETF), spearheaded by Anthropic technical lead Kevin Kelley. By implementing a unified user agent, website owners can now issue a single command to block crawlers specifically tasked with training generative AI models.

Under the new standard, operators can insert a specific user agent—__usage_ai-training—into their robots.txt files to restrict data collection. The directive allows for granular control, enabling site owners to permit access to specific directories, such as press kits, while barring broader scraping. Google confirmed it will interpret this signal as a directive for its 'Google-Extended' agent, which gathers materials for Gemini model development.

Currently, website administrators must identify and configure individual crawlers for every AI provider, a process that is both time-consuming and prone to error. This new protocol aims to replace that fragmented approach with a common signal. However, the companies noted that the directive is not retroactive; it will not remove data already ingested by AI models from previous crawling activities.

Anthropic has already integrated support for the new directive. Microsoft and Google have scheduled their implementation for early 2027, with Microsoft specifically targeting February 1. OpenAI has committed to adopting the signal within the coming months. While this change provides publishers with more authority over their digital assets, it does not affect standard search engine indexing, ensuring that websites remain discoverable in Google Search results.