OpenAI, Google, Microsoft, and Anthropic are backing a common signal in robots.txt that is intended to help website operators communicate preferences about the use of their content for foundational generative-AI training.
Why publishers want a shared signal
Website owners can already use robots.txt to communicate crawler preferences, but AI providers have historically used separate crawler identities and rules. OpenAI says the common signal could reduce the work of maintaining individual instructions for each participating provider.
For publishers, the practical appeal is administrative rather than magical. A single preference is easier to document, review, and update than a growing list of provider-specific rules. The proposal also gives participating providers a common convention to implement instead of asking each publisher to learn a different mechanism.
Adoption will be staggered
OpenAI says it will implement the signal in the coming months. Independent reporting from Nieman Lab says Anthropic is already supporting it, while Google and Microsoft have indicated that their support will arrive next year. The different timelines mean that a publisher cannot assume one rule will have the same effect across every provider immediately.
The signal is voluntary and depends on providers honoring it. OpenAI also says the new mechanism will not replace its existing crawler controls. That makes it better understood as an additional common preference signal, not a universal technical block on every form of crawling or data collection.
What the signal does not settle
A robots.txt preference does not answer every policy question around AI training. It does not by itself define licensing terms, establish ownership of all online material, or determine how a provider should treat content obtained through other channels. It also does not guarantee that an AI system will forget material already used in training.
Implementation details matter. Publishers will need to know the exact token, which crawlers recognize it, whether the preference applies to all content or only selected paths, and how providers distinguish training from search indexing, retrieval, safety analysis, or other uses. Those details are still evolving as the work continues through the Internet Engineering Task Force’s AI Preferences working group.
What website operators can do now
- Document which AI crawlers your current
robots.txtrules address. - Separate preferences for training, search discovery, and operational access where your provider controls support those distinctions.
- Monitor provider announcements rather than assuming simultaneous adoption.
- Keep records of policy changes and review them with legal or editorial stakeholders where content licensing matters.
The common signal is a modest but useful standardization step. It could lower the operational burden for publishers, but its effectiveness will be determined by implementation coverage, clear semantics, and whether providers consistently respect the preferences they say they support.
Sources: OpenAI: Making it easier for websites to express AI training preferences and Nieman Lab’s independent report.
Related: Cloudflare adds traffic analytics and access rules for individual AI agents.
Feature image: AI-generated conceptual illustration for Guava Insights.

