Most practice owners think robots.txt blocks AI assistants from using their data for training. It doesn't. Robots.txt only controls whether crawlers can access your website at all. If you block GPTBot or ClaudeBot there, they can't read your pages, so your practice doesn't appear in ChatGPT or Claude answers. That is the real problem for most practices, not training concerns.
Most practice owners I talk to believe robots.txt is a firewall against AI assistants. Block the crawlers there and ChatGPT, Claude, and Gemini will not touch your data. Google's Gary Illyes just published a video clarifying what robots.txt does: it controls whether a crawler can read your pages at all. It has nothing to do with what happens to your content after that crawler retrieves it. And that distinction breaks how most practices approach their websites and AI discovery.
The Mistake That Kills Visibility
If your robots.txt blocks GPTBot or ClaudeBot, those assistants cannot read your site. Your practice does not appear in their answers. A patient asks ChatGPT for a dentist near them, and you are not in the response. Not because you made a choice about your data. Not because you prioritized privacy. But because the bot could not access your website in the first place.
I see this happen constantly. Someone copies a robots.txt from a template without reading it carefully. Or a security vendor puts blanket restrictions in place and nobody revisits them. Or a site was built years ago with settings that made sense at the time but have never been updated. The practice owner thinks they are being protective. They are actually invisible to AI assistants.
This is the problem that matters. Not whether your content ends up in training datasets. Whether your practice shows up when a customer asks ChatGPT where to go.
What Illyes Explained
RFC 9309, the internet standard that defines robots.txt, describes it as an access preference protocol. It specifies which URLs a crawler may request and nothing more. After the crawler downloads your content, what happens to it falls entirely outside the scope of robots.txt. If you want to block a particular AI crawler completely, you have to name its user-agent token explicitly in your robots.txt file.
GPTBot is the user-agent for ChatGPT. ClaudeBot is Claude. If your robots.txt does not mention them, or if it says Disallow for their paths, they cannot crawl you. The distinction between crawling access and downstream use is what Illyes was clarifying. For years, site owners have conflated the two. They think robots.txt controls what happens to their data after someone reads it. It does not. It only controls who gets to read it in the first place.
Training is Separate, and More Limited
Google does offer one standard mechanism to separate crawling from training: Google-Extended. If you add a rule for Google-Extended in your robots.txt, you can tell Google: crawl my site for search results, but do not use my content to train Gemini or ground it in Gemini Apps. This is a usage control, not an access control. Google still crawls you. Your practice still appears in Google Search. But Gemini does not learn from you.
For other AI companies and crawlers, there is no universal standard yet. The IETF has drafted a proposal that would add a Content-Usage rule to robots.txt, allowing any site to say 'do not train on this content.' That draft is not finalized. It may never be. For now, if you want to keep your content out of AI training for platforms other than Google, your only lever is to block the crawler entirely, which also prevents that AI assistant from seeing your practice at all.
So you are stuck with a choice: be visible to AI assistants, or try to hide from training. You cannot easily do both, unless you only care about Google.
How I Handle It
I know this because I publish a content site automatically and I want AI assistants to find it. My robots.txt explicitly names GPTBot, OAI-SearchBot, ChatGPT-User, and ClaudeBot and allows them. It is easy not to think about it. Someone hands you a robots.txt or you inherit one from a previous site. You do not notice that it blocks crawlers by default. But if the crawler cannot read the site, the assistant has nothing from you to work with. It fills the gap from other sources, or leaves you out of the answer entirely.
So I check my robots.txt on purpose. I allow the crawlers I want to see me. And I go further: I publish a plain-text file that explains who we are and what we do, so AI tools have the full picture in our own words and do not have to guess by piecing together headings and buttons from the page layout.
What to Check in Your Robots.txt
If you run a practice or a service business and you want to appear in ChatGPT and Claude answers when customers ask about your business, go look at your robots.txt. It is at yoursite.com/robots.txt.
- Search the file for 'Disallow: /' at the top level. If that rule exists with no Allow statement for specific crawlers, the bots cannot read your site.
- Search for 'GPTBot', 'ClaudeBot', 'OAI-SearchBot', 'ChatGPT-User'. If you do not see them, or if a Disallow rule blocks their paths, they are blocked.
- If those crawlers are blocked and you did not choose it deliberately, that is your problem right there. Not data privacy. Not training concerns. Simple invisibility.
If they are blocked, change it. Allow those user-agents for the paths you want to be discoverable. You made an access choice without realizing you made it.
The Real Question
Maybe you do want to block all AI crawlers. That is a legitimate choice. But most practice owners I speak with have not made that choice. They have inherited a robots.txt they did not write, or they are following advice that assumed every bot is a threat. They think they are being careful about their data. They are actually being careful about remaining unknown.
The training question is separate and less powerful than most people believe. If you care about whether Gemini learns from your content, add a Google-Extended rule. If you want to express a usage preference for other AI platforms, you can write it today in the way the IETF draft describes, though no crawler is required to honor it yet. But none of that matters if the crawler never sees your site.
The Close
Illyes's clarification shifts the conversation. For years, site owners have worried about robots.txt as a privacy shield. It is not. It is an access gate. Open it to the crawlers you want and then you can negotiate what they do once they are there. Block them at the door and you do not get a negotiation. You get obscurity. And for a dental practice or a service business, that is a much bigger problem than any question about training data.
