10 min read - Shared AI Training Preferences: Decide Before Editing robots.txt

Website owners may soon be able to state one AI-training preference instead of maintaining a separate rule for every participating provider. On 30 October 2026, OpenAI, Google and Microsoft each announced support for a shared signal in robots.txt. The mechanism uses a policy token named __usage_ai-training.

The file edit is small, but the decision behind it is a content policy with technical consequences. Training, crawling, search indexing and grounding are different uses; the proposed signal covers only one of them. It is also an early Internet-Draft, and the three providers have different implementation dates and product-specific rules.

For a startup or SME, the work starts before engineering adds any lines: decide which content may be used for which purpose, who has the right to decide, and how you will test that the published policy still matches that decision.

What was announced, and what is not live yet

OpenAI's 30 October announcement says it will honour the common signal for foundational generative AI training. The company aims to implement it in the coming months; its existing crawler controls will remain in place. The announcement therefore records an intended change, not proof that the shared token already controls OpenAI systems today.

Google's Search Central announcement targets support in 2027. Google says that, when a site opts out through the common token and does not specify Google-Extended separately, it plans to interpret the choice as a Google-Extended disallow. A site can still publish a provider-specific Google rule alongside the broader preference.

Microsoft gives firmer dates in its Bing publisher guidance. It plans to start honouring the new preferences as it crawls from 1 February 2027. Sites that currently use NoArchive or NoCache for Microsoft training or grounding choices have a transition ending on 1 March 2027. Microsoft also introduces separate, page-level grounding settings; those are not part of the shared training token.

These are three primary-source commitments, but they do not establish one activation date or identical scope across providers. A change deployed now must coexist with today's controls until each provider documents that its implementation is active.

The draft expresses a preference, not a network block

The mechanism appears in the IETF Datatracker as Machine-Readable Training Preference Signal, dated 30 October 2026. The Datatracker labels it an individual Internet-Draft with no formal standing in the IETF standards process. It may change, be replaced or expire; the current version is due to expire on 3 May 2027. Its author, Kevin Kelley of Anthropic, presented it on the IETF AI Preferences mailing list as input to the working group's attachment work and said Anthropic has started implementing support alongside its existing crawler controls.

Its example is short:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

Under the draft, this states a preference against generative-AI training for the whole site and a preference in favour of it under /press-kit/. The familiar Allow and Disallow syntax is reused, but the token names a purpose rather than a crawler. A wildcard User-Agent: * group does not match it, and omitting the policy-token group expresses no preference.

The draft also states that the token is not a directive for crawling behaviour. In our reading, a provider could therefore keep fetching a page for search while honouring a preference against using it to modify the learned parameters of a generative model. If a site also wants to restrict access by a named crawler, it needs the relevant crawler rule.

Private agreements may override the preference under the current draft. That clause is another reason to review contracts, licences and platform terms instead of assuming one public file settles every relationship.

Separate four decisions before touching production

A single “AI opt-out” checkbox collapses questions that need different owners and evidence.

Marketing may want qualified pages to remain discoverable. Legal or procurement may find that the organisation does not own all rights needed to grant training use. The web team knows which paths are public, while product owners know whether customer or partner material appears under them. None of those views is enough alone.

Build a small rights map before choosing syntax. For each content class, record its owner, source, governing terms, public paths, intended search treatment, training preference and grounding preference. Include translated pages, uploaded documents, user-generated content, partner material and archives. A French translation does not create new rights, and a public URL does not prove permission for every downstream use.

Our guide to building the evaluation asset before model training uses the same discipline on the input side: access, rights and permitted purpose must be recorded separately.

Choose policy by content class, not by provider name

Provider lists change. Business intent should survive those changes. Start with the content classes the organisation can defend.

A public product manual may be intended for broad discovery and citation, yet its owner may decline model training. A press kit may have terms designed for reuse. A customer portal should already rely on authentication rather than voluntary crawler preferences. A community forum may mix company copy with material supplied under different terms.

Those differences suggest a path policy such as “decline training by default, permit the approved press area, and keep search and grounding decisions separate.” They do not prove that this is the right policy for every business. The organisation must decide after checking ownership, commercial goals and existing agreements.

Avoid two common shortcuts. Blocking every crawler can remove legitimate search access without proving that previously collected content is deleted. Allowing every crawler because search traffic matters can grant a broader operational signal than the owner intended. The new token is useful because it can separate one usage choice, but only participating consumers that implement it are expected to read it.

Treat robots.txt as governed production configuration

Many sites generate robots.txt from application code, an SEO plugin, a CDN rule or a deployment template. A manual edit in one console may be overwritten on the next release. The policy therefore needs a source of truth and a test.

Use a narrow change process:

1. Inventory every place that can emit or rewrite robots.txt, page meta tags and X-Robots-Tag headers.
2. Approve a path-by-purpose matrix with named content, technical and rights owners.
3. Keep current provider-specific rules while announced shared support remains pending, unless the responsible owner chooses otherwise.
4. Add automated checks against the deployed file and representative page headers; test both the production host and the repository fixture.
5. Set review dates around provider implementation milestones and any change to the Internet-Draft.

The test should detect more than missing text. Parse the file with the RFC 9309 longest-match rules that the draft keeps, check sensitive paths, and flag conflicting generated groups. Confirm that search-critical pages retain the intended crawl rules. For Microsoft, sites using the legacy NoArchive or NoCache treatment need a separate transition check before March 2027.

Keep a rendered snapshot and approval record with each release so the policy can be audited later.

Measure what changes after implementation

The shared signal does not promise deletion, payment, traffic or inclusion in an AI answer. Microsoft says allowing grounding makes content eligible to inform its AI responses, and that neither the training nor the grounding choice guarantees inclusion, traffic or payment. Google's crawler documentation states that Google-Extended does not affect inclusion or ranking in Google Search. OpenAI has not yet published its implementation details for the shared token.

Measure outcomes that the available tools can actually observe: deployed policy integrity, crawler requests in server logs, search indexing, referral traffic and provider-specific citation reports where available. Do not report “training stopped” from a fall in crawl traffic; the draft separates retrieval from use, and an external site cannot inspect a provider's full training pipeline.

Record unknowns plainly. A useful quarterly review states which providers have documented support, which controls are active, which paths carry exceptions, and which outcomes cannot be verified. This is the same evidence discipline described in our guide to portable AI governance: name the control, owner, evidence and expiry instead of writing a broad compliance claim.

A practical decision for this quarter

Teams do not need to wait passively for a final standard. They can inventory rights, separate the four decisions and prepare a tested configuration now. Publishing the shared token immediately is reasonable only when the responsible owners approve the meaning, existing crawler controls remain coherent and the team accepts that support is still rolling out.

Otherwise, prepare the change behind a reviewed pull request and activate it when the chosen providers document support. Either route should preserve present rules until their replacements are verified in production.

A common token saves repeated configuration. Content ownership, contract review, search strategy and deployment testing remain separate work, and the path-by-purpose policy behind the file is what you will reuse when providers change their rules.

Explore Exceev's AI and engineering services if your team needs to map content uses, turn the decision into tested web configuration or build an evidence record for future provider changes.

Sources and limitations

We opened and reviewed every source below on 2 November 2026. We added the IETF mailing-list message and Google's crawler documentation on 4 November 2026.

• OpenAI's announcement establishes its intended support, current timing and the continued role of its own crawler controls.
• Google's Search Central announcement establishes its 2027 target and planned interaction between the shared signal and Google-Extended.
• Microsoft's Bing publisher guidance establishes its dates, page-level settings, grounding distinction and transition from NoArchive and NoCache.
• The IETF Datatracker record and Internet-Draft define the proposed token, matching behaviour and boundaries. The document is work in progress, not an endorsed IETF standard.
• Kevin Kelley's IETF mailing-list message introduces the draft and states that Anthropic has started implementing support.
• Google's crawler documentation states that Google-Extended does not affect Google Search inclusion or ranking.

We did not test any provider's unreleased implementation, inspect training systems or verify a website's contracts and content rights. Provider commitments, draft syntax and dates may change. The path matrix, change process and measurement plan are Exceev's analysis, not requirements from the cited organisations. This article is technical and operational guidance, not legal advice or proof that publishing a preference creates an enforceable right in any jurisdiction.