OpenAI to honor a shared robots.txt signal for AI training preferences
OpenAI announced it will honor a shared signal in robots.txt that website operators can use to state their preferences about use of their content for foundational generative AI training. The effort is voluntary.
Details
- Shared signal: The signal is linked to an IETF Internet-Draft titled “training-preference-signal” and draws on the IETF AI Preferences working group’s vocabulary work. OpenAI joins other working group participants in announcing it.
- Why: One preference can be read by multiple participating AI providers, so operators need not maintain separate rules for each crawler.
- Timing: OpenAI plans to implement it in the coming months and will publish details then. No specific date is given.
- Existing controls: Current robots.txt crawler controls remain in effect and will still be respected.
- Call to others: OpenAI encourages other AI providers to honor the signal as well.
Outlook
Other participating providers are not named on the page, and implementation details are undisclosed until launch.