Quick answer: On October 30, 2026, Google, Microsoft, OpenAI, and Anthropic announced they will honor a new shared robots.txt “policy token,” __usage_ai-training, built from an IETF Internet-Draft written by Anthropic’s Kevin Kelley. Instead of naming every AI crawler by hand — GPTBot, Google-Extended, ClaudeBot, and whatever ships next — a publisher adds one rule under the new token and every participating company is supposed to respect it as “don’t use this content to train your models.” Most coverage frames this as a clean win for publishers. It isn’t clean for everyone: per the reporting we checked against Google’s own documentation, a site that opts out through the shared token alone also steps out of grounding — the mechanism that lets Gemini Apps cite and link to a page — unless it separately and explicitly allows Google-Extended. Microsoft says training opt-out and AI citation are kept as two different controls. Nobody is live with this yet — timelines run from already-started to 2027 — so there’s time to get the implementation right, but the planning mistake is available today.
Trend watch — published November 1, 2026, in the GEO Lab. We test our own citation data most days; this one is about a policy signal that could quietly decide whether there’s any citation left to measure.
What did the four companies actually agree to on October 30?
The signal is a new robots.txt policy token called __usage_ai-training, specified in an IETF Internet-Draft — draft-kelley-aipref-training-preference-signal-00, dated October 30, 2026 and authored by Kevin Kelley of Anthropic, building on the AI Preferences (AIPREF) working group at the IETF. A publisher adds a group to their robots.txt keyed to that token and lists the paths they don’t want used to train a generative model’s parameters — one rule instead of separately blocking GPTBot, Google-Extended, ClaudeBot, and whatever a new lab ships next quarter. The same day the draft was published, Google, OpenAI, and Microsoft said they’d support it alongside Anthropic. Reporting is explicit that the token is a usage preference, not a crawling directive: it doesn’t stop a crawler from fetching the page, it tells a participating company what it’s allowed to do with what it fetched. Existing named-crawler rules aren’t replaced — the draft states wildcard groups “MUST NOT be interpreted as matching the policy token,” so your current GPTBot or Google-Extended disallow lines keep doing whatever they already do, alongside the new token rather than instead of it.
Why doesn’t blocking AI training automatically protect — or cost — your AI citations?
Because “trained on your content” and “cited in an AI answer” are two different pipelines that happen to share some of the same crawler infrastructure, and the four companies aren’t handling that overlap the same way. Per the reporting we checked against Google’s own wording, a site that opts out purely through the shared token — without a separate rule for Google-Extended — also falls out of grounding in Gemini Apps, the mechanism that lets a Gemini answer cite and link to that page. To keep training blocked while staying eligible for citation on Google’s side, a publisher has to take the extra, explicit step of adding Google-Extended: Allow on top of the new token. Microsoft describes a cleaner separation: it says AI training and AI grounding are two different controls, so blocking one doesn’t move the other. OpenAI and Anthropic haven’t published the equivalent breakdown for ChatGPT’s and Claude’s web-answer features yet. The honest state of this story on November 1: one company’s documented behavior ties the two together by default, one says it doesn’t, and two haven’t said.
Haven’t we seen this exact confusion before?
Yes, twice, in opposite directions. When Apple shipped Applebot-Extended as its AI-training opt-out crawler, Apple explicitly said blocking it would not hurt a site’s regular search ranking — publishers feared a training opt-out would quietly cost them visibility, and Apple’s answer was no. A census of the top 5,000 sites’ robots.txt files found the opposite already happening in the wild: publishers who thought they were only blocking AI training crawlers had, by naming the wrong user-agent, also blocked the crawlers that feed AI Overview and ChatGPT citations. This week’s story is the first case where the ambiguity isn’t a publisher’s mistake — it’s built into how one company has chosen to wire the new token by default. Same meta-pattern we’ve logged across AI Mode’s ad rollout, the Connected Apps Platform, and Perplexity’s Computer agent this quarter: a new door opens, and the thing that gets you through it is never the thing the headline implies.
What should a publisher actually check before touching robots.txt for this?
Don’t add __usage_ai-training: Disallow as a blanket “block AI, keep my citations” move. On Google’s current documented behavior, that token alone also opts you out of Gemini Apps grounding. If you want training blocked but citation kept, add the extra, explicit Google-Extended: Allow line on top of it — and re-check Google’s live documentation once this actually ships in 2027, since specifics can change before then.
Don’t assume Microsoft, OpenAI, and Anthropic behave like Google here. Microsoft says training and grounding are separate controls on its side. OpenAI and Anthropic haven’t published the equivalent mapping for ChatGPT search or Claude’s web answers — treat their rollout as unverified until each documents its own split.
Keep your existing named-crawler rules. The token is additive, not a replacement — GPTBot, Google-Extended, ClaudeBot, CCBot, and whatever else is already in your robots.txt keeps doing what it does. An audit now has to check both the old named groups and the new token group for conflicts.
Treat this as a voluntary agreement, not a web standard, until proven otherwise. The IETF Internet-Draft has no formal standing in the standards process — it’s four companies agreeing to honor it on their own timelines, from already-started (Anthropic) to 2027 (Google). Compliance is self-reported for now, the same caution we apply to llms.txt adoption claims.
Frequently asked questions
What is the __usage_ai-training robots.txt token?
It’s a new robots.txt policy token, specified in an IETF Internet-Draft authored by Anthropic’s Kevin Kelley on October 30, 2026, that lets a publisher opt a set of paths out of AI model training with one rule instead of naming every AI crawler individually. Google, Microsoft, OpenAI, and Anthropic have all agreed to honor it, on different timelines running from already-started to 2027.
If I opt out of AI training, does that also opt me out of being cited by AI Overviews, Gemini, or ChatGPT?
It depends on which company. Per the reporting we checked against Google’s own documentation, using the shared token alone on Google also removes a site from grounding — the mechanism behind citation — in Gemini Apps, unless the site separately allows Google-Extended. Microsoft says it keeps AI training opt-out and AI grounding/citation as two separate controls. OpenAI and Anthropic haven’t published the equivalent mapping for ChatGPT search or Claude’s web answers yet, so treat that as unconfirmed rather than assuming either answer.
When does this actually take effect?
Nobody is live with it as of this writing. Anthropic says it has already started implementing support. OpenAI says it will support it “in the coming months.” Microsoft set a start date of February 1, 2027. Google said it plans to honor it starting in 2027, without a firmer date reported.
Is this an official, binding web standard?
No. It’s an IETF Internet-Draft that has not been endorsed by the IETF and has no formal standing in the standards process. The four companies’ commitment is a voluntary agreement to honor the token on their own products and their own timelines, not a ratified standard that binds every AI company or that any third party can enforce.
Leave a Reply