On October 30, 2026, four of the biggest AI developers agreed on one short line a website can put in robots.txt to say no to AI training. Google, Microsoft and OpenAI each published support for it the same day, and Anthropic's Kevin Kelley wrote the specification it is built on. The mechanism is an ordinary robots.txt group, but instead of naming a crawler on the User-Agent line, it names a policy token: User-Agent __usage_ai-training, with Disallow / underneath. The token comes from the IETF AI Preferences working group, known as AIPREF, and the draft is draft-kelley-aipref-training-preference-signal-00. The point is that one preference reaches every participating provider at once, instead of a separate rule per company, and it is a preference about training, not a crawler block: it does not stop any crawler from fetching a page, and it does not say anything about whether a page can be cited in an AI answer.
The dates, and the catches that matter more than the dates
Nothing reads the token in production yet, and the start dates are spread out. Microsoft starts honoring the new preferences on February 1, 2027 as it crawls, with March 1, 2027 as the transition deadline for its legacy NoArchive and NoCache signals in this area. Google aims to support the control in 2027, with its crawler documentation saying early 2027. OpenAI says it will honor a shared signal in the coming months, with no date. Anthropic says it has already started implementing support in its crawlers. The Google catch is the one to get right before you edit anything: when a site opts out through the shared token and does not name Google-Extended, Google plans to read that as if the publisher had disallowed Google-Extended. That matters because Google-Extended covers more than training: it also covers grounding, the content from the Search index handed to a model at prompt time in Gemini Apps and in Grounding with Google Search on Vertex AI. So a shared opt-out at Google also switches off grounding citations unless you name Google-Extended yourself. Three more limits from the draft itself: a wildcard User-Agent * group does not count as matching the token, private agreements may override preferences expressed through it, and it sets no payment term and no rule for content collected before the line was added. The draft is an individual Internet-Draft, version 00, not endorsed by the IETF and with no formal standing, and it expires on May 3, 2027 unless revised or adopted.
What to change in your robots.txt now
Decide training and citation separately before you touch the file, because they are separate decisions and the shared token only speaks to training. If you want your pages out of training but still citable in Gemini answers, say so explicitly with your Google-Extended rule instead of letting the shared token decide it for you. Keep the per-company controls you already have until each provider says the shared token is live: Google-Extended is still the control Google reads today, and the same applies to the other companies' own signals. Add the shared group now, so it is already in place when honoring starts, and allow specific paths back in with Allow lines if part of the site, such as a press kit, should stay trainable. Then check the file after February 1, 2027, when Microsoft starts, and again when Google's early-2027 support lands, because a draft this young will move. Our Gemini Decisions API explainer covers the cheap, fast judge Google is building into the same ecosystem, and our Gemini 4 Carbon explainer covers the flagship model your content could end up training. If you want the robots.txt, the crawler controls, and the monitoring around your site checked by someone who runs infrastructure for a living, book a free infra audit with DevOpsBY.
Disclaimer: This post is educational only, not financial or legal advice. Figures are as reported by the sources on 2026-10-30 and may change - always confirm version numbers, requirements, and prices in the official docs before acting on them.
