What changed on October 30, 2026
On October 30, 2026, OpenAI said it will honor a common signal in robots.txt so websites can state a preference about use of their content for foundational generative AI training. The idea is deliberately simple: instead of negotiating a separate opt-out with every provider, a site operator can publish one preference in one familiar file and have it read by multiple participating providers at once. In its OpenAI announcement, the company framed the effort as voluntary, giving publishers a clearer way to communicate how they would like their content to be treated for training purposes.
Reports of the joint announcement say Google, Microsoft and Anthropic are backing the same shared signal, which is what makes this development different from earlier, company-by-company controls. Publishers have long been able to address individual crawlers, but that approach required keeping track of each provider’s documentation, product names and rules. A shared signal, if it is widely implemented, reduces that duplication and gives smaller sites the same practical choice that large publishers have negotiated for themselves. The Unite.AI report on the announcement sets out the joint backing and the reasoning behind a common approach.
OpenAI said it aims to implement the signal in the coming months, that existing robots.txt crawler controls will continue to apply, and that it will share implementation details later. That timetable matters for planning: the announcement is a commitment to honor a preference, not an overnight switch. Until the implementation details arrive, website owners should treat the new signal as an addition to, not a replacement for, the controls they already use.
For readers, the change is less visible but still meaningful. When publishers can express a training preference once and have it respected across providers, the choices behind the content they read become more consistent. It does not settle every question about AI training, copyright or compensation, but it does move the practical conversation from a patchwork of private settings toward a shared convention that ordinary site owners can actually use.
How the shared signal works
The shared signal is a policy token, __usage_ai-training, written as a User-Agent line in robots.txt. To opt a whole site out, a site owner adds a group such as User-Agent: __usage_ai-training followed by Disallow: /. To allow selected material back in, the same group can add Allow lines, for example Allow: /press-kit/, so that a press kit or another public section can remain available for training while the rest of the site is opted out. The token therefore expresses a preference about training use, in the same file and format publishers already maintain for crawlers.
Importantly, it introduces no new robots.txt directive names or grammar. It reuses ordinary robots.txt groups, longest-match precedence and path rules from RFC 9309, the specification that standardizes how robots.txt files are parsed. That design choice keeps implementation straightforward for providers and predictable for site owners: the most specific matching rule wins, groups are read in the usual way, and paths are matched under the familiar robots.txt rules rather than a brand-new syntax that would need to be learned from scratch.
Two details are easy to get wrong and worth stating plainly. A wildcard User-Agent: * group must NOT be treated as matching the token, so a site that blocks all crawlers with a wildcard has not automatically expressed a training preference under this scheme. And if no group contains the token, no preference has been expressed at all. Silence, in other words, is not an opt-out: the signal only exists where the token itself appears in a robots.txt group.
That precision is the point. By tying the preference to a distinct token rather than to general crawling rules, providers can distinguish a deliberate training choice from ordinary traffic management. A publisher can welcome search crawlers, allow AI systems to fetch pages for answering questions, and still say no to foundational training, or the reverse, without those choices collapsing into one another.
What the signal does not do
It is a training-use preference, not a crawling directive. It does not by itself stop a crawler fetching pages, it does not affect search indexing, and it does not remove content already collected or already trained into models. A site that adds the token should not expect its pages to disappear from search results, nor should it expect existing datasets or trained models to be unwound. Those are separate questions, governed by separate rules and, in some cases, separate legal or commercial discussions.
Content-use preferences and traffic-management rules are independent, and confusing the two is the most likely source of disappointment. Blocking a crawler to save bandwidth is not the same as expressing a training preference, and expressing a training preference is not a substitute for blocking a crawler. Publishers need both kinds of control, and they should check that each rule in their robots.txt file is doing the job they intend it to do. The signal is also not a security control: it does not protect passwords, accounts or credentials, a point underlined by recent reporting on Black October API key breaches, which shows why access credentials need their own protections regardless of any robots.txt setting.
Private agreements may override the preference. If a publisher has separately licensed content for training, or has agreed different terms with a provider, that agreement can take precedence over the general signal in robots.txt. Site owners with commercial partnerships should therefore check their contracts before assuming that a single file change settles their position.
Finally, the status of the underlying document should be kept in perspective. It is an individual IETF Internet-Draft, draft-kelley-aipref-training-preference-signal-00, by Kevin Kelley of Anthropic, dated October 30, 2026, expiring May 3, 2027, not an endorsed IETF standard. The IETF Internet-Draft should therefore be treated as work in progress: its details may be revised, its expiry date may matter if it is not renewed, and its adoption depends on providers choosing to implement it rather than on any standards body mandating it.
When each company says it will apply
The timelines are not uniform, and that unevenness is part of the story. Anthropic is reported as already adopting or implementing the signal, which fits its role in authoring the draft. OpenAI, as noted above, aims to implement it in the coming months and has promised further implementation details. For publishers, the practical consequence is that the same robots.txt file may be honored at different times by different providers, even though the preference itself is written only once.
Microsoft is reported as applying the signal from February 1, 2027 as it crawls, with separate training and grounding controls. That distinction matters: training a foundation model and grounding answers in retrieved content are different uses, and Microsoft’s approach appears to keep them separate rather than treating one preference as covering both. Publishers who care about one use but not the other should watch for those separate controls when they are published.
Google is reported as implementing the signal in 2027, described as early 2027 in its crawler documentation. Given Google’s central role in search and crawling, its timetable will be closely watched, but it also reinforces the need to keep search and training preferences distinct: a change for training should not be expected to alter search visibility, and vice versa.
One caution is essential. Meta was reported as expressing support for a common signal but not confirmed as honoring this token, so Meta should not be claimed as a participant in this particular scheme. Listing a company as a supporter of the general idea is not the same as confirming that it will read and honor __usage_ai-training, and publishers should not assume coverage that has not been confirmed.
A practical checklist for website owners
Start by deciding whether the preference should be site-wide or per-folder. A site-wide Disallow: / is the simplest choice for publishers that want a blanket training opt-out, while Allow exceptions let a site keep specific sections, such as a press kit, open. Per-folder preferences give finer control but require more care: each folder’s rules should be reviewed so that an exception does not accidentally expose more than intended.
Keep existing per-crawler rules, such as those for GPTBot, in place until the shared signal is implemented. Removing a working control in anticipation of a future one would leave a gap, and providers have said existing crawler controls will continue to apply. The shared token should be added alongside current rules, not used as a reason to delete them prematurely.
Test the robots.txt syntax before relying on it. A misplaced group, a typo in the token or a wildcard group mistaken for a token group can silently fail to express any preference. Use a robots.txt tester, check that the token group is parsed as intended, and confirm that Allow exceptions behave under longest-match precedence as expected. Remember throughout that search visibility is separate from training preference: changing a training setting is not a way to manage search indexing, and search settings do not substitute for a training preference.
Finally, revisit the file when providers publish implementation details and when the draft is revised or expires. The draft’s May 3, 2027 expiry date is a natural review point, and each provider’s implementation notes may add provider-specific details or separate controls, such as Microsoft’s training and grounding distinction. A robots.txt file is not a one-time setting; it is a living statement of preference that should be reviewed as the convention around it develops.
Why this matters beyond publishers
For small publishers, the benefit is practical: less duplicated work. A blogger, a local news site or an independent creator should not need a legal team or a separate negotiation with each AI provider to express a basic preference about training. One rule, written once, lowers that barrier and makes the choice accessible to sites that would otherwise be left out of the conversation entirely.
For readers and creators, clearer choices build trust. When a site’s training preference is stated openly in a standard file, creators can see what a publisher has chosen, and readers can better understand the terms on which content they enjoy is being used. That transparency is especially relevant as AI tools become part of everyday creative work, from the AI coding assistants in 2026 that developers use daily to automated systems such as Google DeepMind CodeMender, which shows how AI agents are increasingly acting on code and content rather than merely reading it.
At the same time, honesty about its limits is essential. The signal is voluntary, its rollout is uneven, and it is not a guarantee or an enforcement mechanism. A provider that has not implemented it will not be bound by it, a private agreement can override it, and content already collected is not affected. That makes it a step toward a common convention, not a settlement of the training debate, and its real test will be whether implementation follows announcement across all the companies that have backed it.
