AI training opt-out: OpenAI, Google, Microsoft and Anthropic agree on one robots.txt rule

Four of the biggest AI companies, OpenAI, Google, Microsoft and Anthropic, said on Friday, 30 October, that they will all respect a single robots.txt rule that tells them not to use a website’s pages to train AI models. Until now a site owner who wanted out had to block each company’s crawler by name, and keep up as new ones appeared.

The new token is called __usage_ai-training. A site adds it to the robots.txt file at its root, and the four companies have promised to treat it as a training opt-out.

How does the AI training opt-out work?

Robots.txt is the plain text file every website can keep at /robots.txt to tell crawlers what they may visit. The new rule uses it like this:

User-agent: __usage_ai-training
Disallow: /

That blocks training on the whole site. A publisher can also block only some sections, or add Allow lines for parts it is happy to share.

The idea comes from a draft submitted to the IETF, the body that writes many internet standards, by Kevin Kelley of Anthropic. It is an individual draft, so it has no official standards status yet; for now it works because the four companies have said they will follow it.

When does it start?

The timelines are different:

• Anthropic says it has already started implementing it.
• OpenAI says it will support it in the coming months.
• Microsoft has given a date: 1 February 2027.
• Google says early 2027. Where a site uses the new token, Google says it will treat it as if the site had blocked Google-Extended, its existing AI training control.

What it does not do

This is where people may get the wrong idea.

• It is about training, not visiting. Crawlers can still fetch pages for search or for an AI answering a live question. Microsoft keeps “grounding”, where its AI reads a page to answer a user, as a separate setting with its own page-level control.
• It does not delete anything. Content already collected into datasets, or already inside trained models, stays there.
• It is a request, not a lock. The signal states the site owner’s preference. Paid licensing deals between a publisher and an AI company can override it, and companies that never signed up are not bound by it.
• The old controls stay. Blocking named crawlers such as GPTBot or Google-Extended still works.

What it means for Pakistan

For Pakistani news sites, bloggers and the freelancers who build and run websites for clients abroad, this turns a messy job into one line. A WordPress developer in Lahore or Karachi who is asked “can we stop AI using our content?” will soon have a simple, standard answer, and it is worth adding to client checklists now, even before Google and Microsoft switch it on.

The catch is the trade-off. Our own analysis: blocking training is unlikely to hurt a site’s normal Google ranking, because search crawling is handled separately, but sites should check each company’s documentation before changing anything, since the rules for AI answers and summaries are still being written.

The next date to watch is 1 February 2027, when Microsoft says its support goes live, with Google expected around the same time.