The shared AI training opt-out: your Azure robots.txt plan

On Friday 30 October 2026, Google, Microsoft and OpenAI each published a post saying they will honour a single robots.txt group whose User-Agent line reads __usage_ai-training. The same day, an Anthropic engineer posted the IETF draft that defines it. Two lines in one file now tell four AI labs whether your site may be used to train their models. For anyone running a site on Azure App Service, Static Web Apps or Front Door, the question is no longer "which bots do I list" but "which posture do I want". Microsoft has attached dates to the answer: the NoArchive and NoCache tags that Bing has read as an AI signal since 2023 stop counting for training on 1 March 2027.

What each company committed to

The four announcements are not identical. They share a token, and they differ on timing, on what else the token implies, and on whether grounding is covered at all.

Microsoft's calendar is the one with teeth. Temporary grounding preferences open in Bing Webmaster Tools on 15 December 2026, but only for sites that already use NoArchive or NoCache on that date. The new preferences take effect as pages are recrawled from 1 February 2027. Sites that have not published a new training preference by 1 March 2027 lose the restriction the old tags gave them. The temporary settings expire on 30 September 2027.

How the token actually works

The draft calls __usage_ai-training a policy token. It sits where a crawler name normally goes, so every robots.txt parser that follows RFC 9309 already accepts it. No new directive, no new file. The minimal form is:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

That expresses a preference against training on every path except the press kit. Path matching follows the normal robots.txt rules, so you can carve out documentation you want models to learn from while reserving everything else.

Four properties of the draft matter more than the syntax.

• The wildcard group does not count. The draft says User-Agent: * groups "MUST NOT be interpreted as matching the policy token". If your robots.txt only has a wildcard group, you have expressed no training preference at all, whatever its Disallow lines say.
• It is a preference, not a crawl directive. Crawlers keep fetching and indexing. Google states that Google-Extended is not a ranking signal, and Microsoft states that traditional Bing rankings are unaffected. The token changes what a participating lab may do with content after it has been fetched.
• "Training" is narrowly defined. The draft borrows the IETF AIPREF vocabulary: using an asset "to modify the learned parameters of a generative AI model". Retrieval at prompt time, grounding, summarisation on a user's behalf and search are all outside it.
• Vendor tokens may win. The draft lets a consumer treat its own proprietary token as more specific than the shared one. Google has already said it will: if you explicitly allow Google-Extended and disallow the shared token, Google reads the explicit allow.

The token is also a bridge, not the destination. The IETF AIPREF working group has two drafts further along: a vocabulary (draft-ietf-aipref-vocab-08, with the labels train-ai, ai-use and search) and an attachment mechanism (draft-ietf-aipref-attach-05) that carries those labels in a Content-Usage HTTP header or a Content-Usage: line in robots.txt. Cloudflare has shipped a third dialect since September 2025, Content-Signal: search=yes, ai-train=no. None of the four labs has said it reads either of those today. The shared token exists because it works with unmodified parsers now, while the working group finishes.

What it does not cover

Grounding and retrieval. This is where the vendors split, and where a publisher's revenue question lives. Microsoft gives grounding its own preference with three values: Allow, DisallowExtended (content may ground Microsoft-operated experiences only) and Disallow. It cannot be set in robots.txt; it must go on the page. Google folds grounding into Google-Extended, so there is no Google setting that says "ground my pages in Gemini but do not train on them". Sites that allow Google-Extended remain eligible for Gemini Apps answers and the traffic those send. OpenAI keeps retrieval on OAI-SearchBot and ChatGPT-User. If grounding is the thing you care about, the shared token alone does nothing for you.

Everyone who is not one of the four. Meta, Perplexity, Apple and Common Crawl have not signed up as of this writing. Apple's Applebot-Extended, Common Crawl's CCBot and the rest still need their own groups. So does any crawler that ignores robots.txt, which the token cannot help with. Bot management at Front Door or Application Gateway remains the tool for those.

The past. Anthropic's crawler page talks about excluding a site's "future materials" from training sets. Nothing in any of the four announcements is retroactive.

Your own model. The token tells the four labs what to do. It says nothing about the Azure OpenAI deployments or fine-tuning jobs your own organisation runs, which is the point of the Swedish and EU section below.

Decision guide: three postures

Most sites fall into one of three postures. Pick one deliberately, write it down, and let the robots.txt follow.

Posture A: allow training. Typical for B2B companies whose content exists to be found and cited. Technspire runs this posture: our own app/robots.ts explicitly allows GPTBot, ClaudeBot and Google-Extended because we want assistants to learn what we do and recommend us. Under the new draft, that implicit allow is now worth making explicit, because an absent group means "no preference" and an explicit allow survives any future change of default.

Posture B: refuse training, keep grounding and search. Typical for publishers and documentation sites whose business is traffic. The shared token says no to training. For Bing, leave grounding at its default. For OpenAI, keep OAI-SearchBot allowed. For Google, you face the gap above: you either accept that the implied Google-Extended disallow also removes you from Gemini grounding, or you add an explicit Google-Extended allow and accept Gemini training as the price of Gemini traffic.

Posture C: refuse both. Typical for paywalled, licensed or personal-data-heavy sites. Shared token disallow, Bing ai-grounding:disallow on every page, explicit Google-Extended disallow, plus per-bot groups for the non-participants.

Shipping it on Azure and Next.js

1. The robots.txt group

In a Next.js App Router project the file is app/robots.ts. The policy token is just a user-agent string, so no special support is needed; the rules array has accepted arbitrary agents since Next 13.3. A posture B file looks like this:

import type { MetadataRoute } from 'next';

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      { userAgent: '*', allow: '/', disallow: ['/api/', '/admin/'] },
      // Shared AI training preference (draft-kelley-aipref-training-preference-signal)
      { userAgent: '__usage_ai-training', disallow: '/', allow: '/docs/' },
      // Keep search and user-directed fetching open
      { userAgent: ['OAI-SearchBot', 'Claude-SearchBot', 'ChatGPT-User', 'Claude-User'], allow: '/' },
      // Non-participants still need their own groups
      { userAgent: ['CCBot', 'Applebot-Extended'], disallow: '/' },
    ],
    sitemap: 'https://example.se/sitemap.xml',
  };
}

Keep the wildcard group's Disallow lines for crawl hygiene; they do not carry a training meaning. If you want posture A, change the policy token line to allow: '/' and drop the disallow.

2. Bing page-level tags

Grounding for Bing can only be set per page. In Next.js the metadata other field emits plain name/content meta tags, which is what Bingbot reads:

// app/layout.tsx
export const metadata = {
  other: {
    Bingbot: 'ai-grounding:disallow',   // or 'ai-training:disallow' for page-level training
  },
};

Check the exact value strings against the Bing Webmaster Tools help page before you ship, and put the tag in the layout that covers the pages you mean, not globally by reflex. If you currently rely on NoArchive or NoCache, record the temporary grounding preference in Bing Webmaster Tools when the option opens on 15 December 2026, then migrate before 1 March 2027.

3. Front Door caching and the future header

Two Azure-specific traps. First, if Front Door or the CDN caches /robots.txt, purge that path after deploying, or your new group may sit behind a stale copy for days. Second, if you want to be ready for the AIPREF header before anyone reads it, Front Door's rule set can add it at the edge with a Modify Response Header action; the same result is available from headers() in next.config.js or from globalHeaders in a Static Web Apps configuration file:

// Azure Front Door rule set action (Bicep)
{
  name: 'ModifyResponseHeader'
  parameters: {
    headerAction: 'Overwrite'
    headerName: 'Content-Usage'
    value: 'train-ai=n'
    typeName: 'DeliveryRuleHeaderActionParameters'
  }
}

Treat the header as optional insurance. The robots.txt group is what the four labs have promised to read.

4. Verify

curl -s https://example.se/robots.txt | grep -A2 __usage_ai-training
curl -sI https://example.se/ | grep -i content-usage

Then run the Google Search Console robots.txt report and the Bing Webmaster Tools robots.txt tester. App Service deployment slots have their own hostnames; a staging slot should disallow everything anyway, so do not copy the production file there blindly.

The Swedish and EU angle

The token arrives into a legal frame that already exists in Sweden. Upphovsrättslagen 15 a § gives anyone with lawful access the right to make copies for text and data mining, unless the author "på lämpligt sätt har förbehållit sig" that right. That sentence implements Article 4(3) of the DSM Directive, which for content made available online means a machine-readable reservation. The Swedish preparatory works never said what format counts, and no Swedish court has ruled on it. A robots.txt group that four labs have publicly promised to read is a stronger candidate for "lämpligt sätt" than anything a site owner could point to a week ago. It is still a candidate, not a tested rule.

For the four signatories the legal question matters less, because they have already bound themselves. Article 53(1)(c) of the AI Act requires every general-purpose model provider to run a copyright policy that identifies and respects rights reservations. The Commission's Code of Practice, published 10 July 2025 and signed by Google, Microsoft, OpenAI and Anthropic, turns that into Measure 1.3: employ crawlers that follow robots.txt per RFC 9309, identify and comply with other appropriate machine-readable protocols, engage with rightsholders on standards for Article 4(3) reservations, and publish information about the crawlers used. Read against that measure, Friday's announcements are the four largest signatories delivering on a commitment they made last year, which is also why Google's post cites the Code by name.

Two Swedish-specific points follow. First, Microsoft has not published a product-by-product map from the Bing preferences to Microsoft 365 Copilot web grounding; the DisallowExtended value speaks of "Microsoft-operated experiences". If your organisation's decision about AI tools rests on where Copilot gets its web context, as the Regeringskansliet decision we covered last week did, ask Microsoft for that mapping in writing rather than inferring it. Second, public bodies that publish on .se domains should take the same documented decision a private publisher would. Choosing posture A for öppna data and posture B or C for journalistic or personal-data-heavy material is defensible; having no recorded decision is not.

The last point is the one most Azure teams miss. If your organisation crawls the web to feed a RAG index or a fine-tuning job on Azure OpenAI, you are a consumer of these signals, not only a publisher of them. The data protection side of that was covered in our July piece on the EDPB's scraping rules. Add the copyright side now: make your crawler read the __usage_ai-training group before any fine-tuning run, read Cloudflare's ai-train signal while you are at it, and log what you honoured. The DSM reservation binds non-signatories too, and the Code of Practice is the benchmark a Swedish supervisory authority will hold up when it asks how you identified reservations.

Checklist for this quarter

• 1. Pick a posture per site and record who decided and why. Marketing sites and documentation portals may legitimately differ.
• 2. Add an explicit __usage_ai-training group to every public robots.txt, even for posture A. Absence is "no preference".
• 3. Decide Google-Extended explicitly. Do not let the implied disallow surprise your Gemini traffic in 2027.
• 4. Inventory NoArchive and NoCache. Any site using them has until 1 March 2027 to publish new Bing preferences, with the temporary grounding setting available from 15 December 2026.
• 5. Add Bingbot page-level tags where grounding matters; robots.txt cannot express it.
• 6. Keep per-bot groups for CCBot, Applebot-Extended and others outside the agreement.
• 7. Purge the Front Door cache for /robots.txt after each change and verify with curl.
• 8. If you crawl, honour the same signals in your own Azure pipelines and keep the log.

Conclusion

Four labs agreeing on two lines of robots.txt is a small technical event with a large administrative consequence: there is now a default answer to "did you reserve your rights in a machine-readable way", and it costs ten minutes to give. The work is in choosing the posture, handling the Google grounding gap, and meeting Microsoft's March 2027 date. Do that before the labs start reading the token, not after.