October 30, 2026
A New AI Training Opt-Out Is Coming. Don’t Block Your Own Leads.
Independent B2B publishers have a new choice to watch. The mistake would be treating every crawler as if it does the same job.
A new proposal lets websites express a shared preference about AI model training. That sounds simple until someone edits the wrong crawler rule and a valuable article stops showing up where buyers look for answers.
If your articles bring in consulting inquiries, sponsorship conversations, or newsletter subscribers, don’t turn a traffic decision into a guessing game.
Dear Suzannah
Question: “I publish a small industry newsletter and resource library. I want more qualified readers, but I don’t want every AI company using my work to train models. Can I opt out without disappearing from search?”
Answer: You can review and use current provider-specific controls. The new shared signal is still a proposal, with adoption taking time. More importantly, a rule for training is not the same as a rule for search. Decide what each crawler may do before you touch your website.
Here’s the deal
On October 30, 2026, an IETF Internet-Draft proposed a shared __usage_ai-training policy token in robots.txt. Website owners could express a preference against model training with a familiar-looking rule. The document is a work in progress, not a finalized standard or a guarantee that every company follows it today.
That same day, Google announced plans to support the shared control in 2027. Google says its existing Google-Extended control continues to work, and that opting out through Google-Extended does not itself remove a site from ordinary Google Search.
The catch? Google-Extended also relates to some Gemini grounding uses. Blocking that control can have discovery trade-offs outside standard Google Search. So the right answer depends on where your customers actually find you.
What this actually helps you do
Imagine you run a WordPress resource library for independent manufacturing suppliers. Your comparison articles are responsible for some of your strongest inbound inquiries. You want to protect original reporting while keeping the pages open to readers who are researching vendors.
This review produces three benefits:
Separate ordinary search crawling from AI training preferences before changing access.
Give your editor and website administrator one documented answer about what to allow.
Check whether important resource pages remain discoverable after an approved change.
Review Googlebot access, indexing and search-preview controls.
Review current provider-specific training controls; watch the shared draft.
Review each provider’s published rules. Some controls affect both training and grounding.
Review hosting and security blocks separately from published policy choices.
A single broad block can affect more than the one use you intended. These purposes are related but not interchangeable.
How to review your site, step by step
- Choose a small test group. Pick five public articles that consistently attract relevant readers or inquiries. Record their URLs and purpose.
- Read your current
robots.txt. Find it at the root of your website, usually/robots.txt. Save a copy of what is there. Do not replace existing rules just because a sample looks tidy. - List the crawlers by purpose. Check Googlebot for Google Search and Google-Extended for certain Gemini uses against Google’s current crawler documentation. A training preference and a search access rule are different decisions.
- Write down the actual business choice. Which content should remain discoverable? Which content do you prefer not be used for training? Which AI answer surfaces matter to your inbound demand? Ask for an editorial decision before the technical edit.
- Mark the new shared signal as pending adoption. The proposed
__usage_ai-trainingtoken is worth tracking, but don’t assume it replaces existing provider-specific settings today. Google has announced a 2027 plan, not universal enforcement on day one. - Have a human approve any change. Your website administrator should review the proposed rule, confirm it doesn’t conflict with search access, and keep a copy of the prior version for rollback.
- Test discovery and actual business outcomes. Use Google Search Console’s URL Inspection tool for selected pages. Track search impressions, relevant referral sessions, qualified inquiry submissions, and newsletter signups over time. Search traffic fluctuates, so don’t assign every change to one setting.
This is a review sequence, not a claim that every AI provider supports the new shared token yet.
Tips and tricks
Give your technical team two separate columns labeled Search and AI model training. It prevents the familiar meeting where everyone says “block the bots” and nobody has defined which bots.
Keep a short dated note beside every choice. If a platform changes how it interprets a control next quarter, you will know why the original decision was made.
Common mistakes
- Blocking Googlebot to block model training. That can interfere with ordinary Google Search discovery. Review Google-Extended and related controls separately.
- Assuming the draft is already universal. The October 30 proposal does not make all providers compliant today.
- Forgetting the inbound goal. A technically correct block can still be a poor business choice if it disrupts a channel your best customers use.
- Publishing the rule without checking it. An incorrect path or overly broad restriction can affect more pages than intended.
Human review checklist
- We recorded five high-value pages and their current search visibility.
- We reviewed the live
robots.txtand saved its original version. - We separated search, training, grounding and security concerns.
- We checked current provider documentation instead of treating a proposal as a guarantee.
- A named person approved any proposed change and the rollback plan.
- We scheduled follow-up checks for discoverability and qualified inbound actions.
How to measure success
Start with a simple before-and-after record: number of selected pages accessible to the search crawler, number of relevant organic sessions, inquiries tied to those pages, and inquiries that match your service. If impressions stay flat and qualified requests rise, you have something worth investigating. If a page disappears after a change, review crawl access before celebrating the policy.
For a companion approach to inquiry quality, read AI Search Visibility Is a Lead Signal. Start Measuring It. To make business facts easier to discover, see Your Next Lead May Be an AI Agent.
FAQ
Is the new shared training signal a finished standard?
No. The October 30, 2026 document is an IETF Internet-Draft. Participating providers may adopt it over time, and details can change.
Will Google-Extended block normal Google Search?
Google says Google-Extended does not affect inclusion or ranking in Google Search. It does affect certain Gemini training and grounding uses, which may matter to your publishing goals.
Does robots.txt keep private documents safe?
No. It publishes crawler instructions and is not a security barrier. Confidential files need access controls, not merely crawler preferences.
Glossary
Crawler: Software that fetches pages for search, analysis or other uses. robots.txt: A public file with rules for crawler access. Policy token: A named signal that expresses a specific content-use preference. Grounding: Using retrieved information to inform an AI answer. Qualified inquiry: A request from a potential customer who fits the services you offer.
Sources and further reading
- IETF Internet-Draft: Machine-Readable Training Preference Signal, published October 30, 2026. Proposed policy token and limitations.
- Google Search Central: Supporting a shared AI training control for websites, October 30, 2026. Announced 2027 support and continued provider-specific controls.
- Google Crawling Infrastructure: Google’s common crawlers. Differences between Googlebot and Google-Extended.
- Google: How Google interprets the robots.txt specification. File placement, groups and matching rules.
- Google Search Central: AI features and your website. Search controls and their limits.
One practical next action
Open your site’s robots.txt, save a copy, and identify which rules affect your five most important articles. Don’t change a thing until you can explain which customer discovery channel each rule protects.
Your content has a job to do. Make sure the policy doesn’t get in its way.
