How to block AI training on your website, with the new shared robots.txt signal

The Morse teamPublished November 2, 20267 min read

Here's how to block AI training on your website: add a robots.txt group for the new shared token, User-Agent: __usage_ai-training with Disallow: /, and keep a Disallow: / group for each company's own training crawler (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, and CCBot) until every company reads the shared one. On October 30, 2026, Google, Microsoft, OpenAI, and Anthropic each said they would honor the shared token, but only Anthropic says it has started, and Microsoft's start date is February 1, 2027.

We read the Internet-Draft that defines the token, each company's announcement, and the crawler documentation of OpenAI, Anthropic, Google, Apple, Meta, Common Crawl, and Perplexity on November 2, 2026, plus the help pages of WordPress.com, Squarespace, and Substack. Every line of robots.txt below comes from those pages. It's the first time four AI companies have agreed to read one opt-out instead of making publishers learn a crawler name per company, and they deserve credit for it.

What changed on October 30, 2026?

Four companies agreed to one opt-out line. Anthropic's Kevin Kelley submitted an Internet-Draft, "Machine-Readable Training Preference Signal," that lets a site state a training preference with an ordinary robots.txt group keyed by a "policy token" (IETF). The same day:

• Google said that if a site uses the token to opt out and doesn't name Google-Extended, it will "interpret that as if the publisher had disallowed Google-Extended," and that it aims to support the control "in 2027" (Google). Its crawler documentation says "early 2027" (Google).
• Microsoft said the new preferences take effect "on February 1, 2027, as Microsoft crawls," and that after March 1, 2027 its older NoArchive and NoCache tags "will no longer be read as a preference for training or grounding" (Microsoft Bing).
• OpenAI said it will honor the signal for "foundational generative AI training," aims "to implement the signal in the coming months," and called it a "voluntary effort" (OpenAI).
• Anthropic, through Kelley's post to the IETF working group, said it "has already started implementing support for this draft," while adding that it will "take us some time to add support across our services" (IETF mail archive).

The draft is an individual submission. Its own front page says it is a work in progress that expires on May 3, 2027, and Datatracker lists it as an individual draft with no formal standing (IETF Datatracker).

How do I block AI training with robots.txt?

Put these groups in the robots.txt file at the root of your domain: the shared token first, then the company tokens that work today. Then repeat them on every subdomain you want covered, as Anthropic's help page notes you must (Anthropic):

User-Agent: __usage_ai-training
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: CCBot
Disallow: /

To keep a folder open to training, such as a press kit, add Allow: /press-kit/ under the shared token. The draft uses exactly that example, and longest-match rules decide which line wins for each path.

Three steps finish the job:

1. Load the file in a browser at yourdomain.com/robots.txt and check each group sits on its own lines, with a blank line between groups.
2. Wait a day. OpenAI says it "can take ~24 hours" for its systems to adjust (OpenAI), and Meta says changes may take up to 24 hours (Meta).
3. For a single page on Bing, use Microsoft's page-level setting instead: <meta name="Bingbot" content="ai-training:disallow"> in the page's head, or the header X-Robots-Tag: Bingbot: ai-training:disallow (Microsoft Bing).

Does "User-agent: *" already opt me out?

Not under the new signal. The draft says "wildcard groups (i.e., User-Agent: *) MUST NOT be interpreted as matching the policy token." Kelley's post spells out the result: a robots.txt file that doesn't name the token "expresses no preference at all" (IETF mail archive).

A wildcard block still asks compliant crawlers to stay away, so it keeps working as a crawl rule. But the training preference is a separate statement about use, and you have to make it on its own line. The two top guides we read for this search on November 2, 2026 were written before the signal existed, and one still gives its steps for "Google Bard" (iubenda). Neither warns about the wildcard.

Which AI crawlers train on your site, and which read the shared signal?

GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, and meta-externalagent are the training controls their owners name, and only OpenAI, Anthropic, Google, and Microsoft have said they'll read the shared signal. We read each owner's own page on November 2, 2026.

Blocking Claude-User and Claude-SearchBot doesn't stop training, and Anthropic warns it may reduce your visibility in search. Google-Extended covers more than training. Per Google's documentation it also governs grounding, so a site that disallows it gives up appearing in Gemini Apps answers drawn from Search.

What doesn't robots.txt stop?

Four things, by the companies' own words:

• Crawlers that skip it. OpenAI says that for ChatGPT-User, which fetches pages a person asks for, "robots.txt rules may not apply" (OpenAI). Perplexity says Perplexity-User "generally ignores robots.txt rules" (Perplexity), and Meta says meta-externalfetcher may bypass them. We cover Meta's other crawlers, and the terms behind them, in Meta's commercial terms.
• Content already collected. The draft ties a preference to the robots.txt file "in effect when the asset was retrieved." Squarespace says its AI-crawler setting "doesn't retroactively remove content previously scraped" (Squarespace).
• Private deals. The draft says "private agreements MAY override preferences expressed via this mechanism."
• Anyone who chooses not to comply. Every page here describes a voluntary system. Microsoft adds that "neither choice guarantees inclusion, traffic, or payment."

How do I block AI training on WordPress.com, Squarespace, or Substack?

Each has a switch that writes the rules for you:

• WordPress.com: go to Settings, then Reading, and under Site Visibility tick Prevent third-party sharing. WordPress.com says it "adds known AI bots to the 'disallow' list in your site's robots.txt file" and keeps your content out of its third-party partner network, with the caveat that "it is up to AI platforms to honor this request" (WordPress.com).
• Squarespace: open Settings, then Crawlers, and check Block known artificial intelligence crawlers. It's unchecked by default, and it adds rules for a list of bots that includes GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, and CCBot (Squarespace).
• Substack: open your publication's Settings, scroll to Privacy, and turn on "Tell AI tools not to train their models on your content." Substack says it applies "only to AI tools that respect this setting" (Substack).

None of the three help pages mentioned the __usage_ai-training token when we read them on November 2, 2026. If your platform lets you edit robots.txt, add the shared group yourself. We cover what Reddit does with posts you've already published in does Reddit sell your data to AI, and LinkedIn's own switch in how to opt out of LinkedIn AI training.

How Morse keeps your unpublished writing private

robots.txt only covers what you've already published. The drafts come first, and many writers paste them into an AI chat for a second opinion, where the training question starts over in each app's settings. Our table of which AI companies train on your data shows how many leave it switched on by default.

Morse doesn't train AI on your content, sell your data, or show ads, so there's no training switch to hunt for. Your chat history is stored only in your browser, encrypted, not on Morse's servers. Pick a mode per chat:

• Private, for a draft you haven't published. It runs open-weight models like DeepSeek, Kimi, and GLM, and Morse has zero data retention agreements with its inference, search, and other providers, so they keep nothing.
• E2EE, for writing no one should read before it's out. Your browser checks the enclave's proof that it's genuine, then encrypts the message so only that enclave can read it. Morse's servers relay text they can't read, with the guarantee resting on the enclave hardware and attestation working as designed. E2EE chats come with Pro, at $19 a month.
• Anonymous, for a pass from GPT, Claude, or Gemini. Morse's proxy removes your account, email, and IP address, so the provider sees the draft without knowing it's you. It may keep it for a limited time to check for abuse, under its own policy.

Clearing your browser deletes those chats for good, and Morse can't recover them, so keep your manuscript in your own files. More on working this way is in private AI for writers.

Put the shared token in your robots.txt today, then try Morse free for the drafts, with no card needed.