Most practices and small businesses worry about robots.txt and AI crawlers, but they are solving the wrong problem. I realized that if your website structure is poor or key information lives only in images and PDFs, robots.txt rules do not matter. AI assistants will find nothing useful anyway. What I focus on now is making sure the information AI tools need is there, written as plain text, in a structure they can follow. That comes before worrying about the rules.
I keep watching practices and service businesses read the news about robots.txt and AI crawlers and start worrying about whether they are blocking the right bots or leaving the wrong ones open. It is the wrong worry. I have dealt with enough websites to know that most of them have a bigger problem first: the information is there, but the structure is not.
The robots.txt distraction
The news lately has been about managing crawler access: which bots are for search, which are for training, which should be allowed and which blocked. It is real and important. But I realized something from building websites for practices and small service businesses. When an AI assistant like ChatGPT or Claude visits a website to answer a question about whether to use that business, the problem is almost never the robots.txt file. The problem is that the website makes it hard to find the answer.
I have a distribution business I run. I wanted to know what AI assistants would say if someone asked them about us. So I checked. The answers were thin. They had found pages on the site, but not the pages that explained who we are or what we do. The crawlers could get in. They just did not know what to look for.
That is when I created llms.txt. It is a plain text file we publish alongside robots.txt, and it answers the basic questions about us: who we are, what we do, who we serve, how to reach us. I wrote it in our own words, not hidden inside a website layout where a machine has to piece it together from headings and buttons and images. When an AI assistant visits, it now has a complete picture in one place.
What stops AI tools from finding you
A good robots.txt does two things: it tells search engines to focus their crawl budget on pages that matter, and it tells training bots whether you want them learning from your content. Both are worth doing right. But they assume one thing is already true: that the pages you do want crawled are readable to begin with.
I redesigned the deepconnects website with this in mind. Everything that matters is written as plain text in the page itself. Your calling hours, your services, your process, how to contact you, all of it is text. Not text inside an image. Not hidden behind a login or stuck in a PDF. Text on the page. The reason is simple: a crawler reads exactly what a visitor reads. If I bury information in a graphic or rely on a site layout to tell the story, an AI tool will miss it.
What I see in practices all the time: critical information lives only in images. The logo says the practice name. The hours are on a banner. The list of services is a graphic. If a crawler can even get in (and sometimes robots.txt blocks it anyway), it finds a structure but no substance. The AI assistant comes away with nothing useful to answer a customer question.
What you should check in your own website
Before you worry about whether to allow OAI-SearchBot or block GPTBot, check whether the essential information AI tools need is actually there in readable form.
- Your practice name, specialty, and location are written as text on the page, not just in logos and headers
- Your hours, contact number, and address are plain text, not inside an image or a PDF
- Your services are listed as text, not as a gallery of graphics
- Your office locations (if you have multiple) are described in text that explains them, not shown only in maps or image overlays
- The process for booking an appointment, uploading documents, or getting started is described in words
If those are missing, adding them takes priority over robots.txt rules. You can let every AI crawler in the world through your robots.txt, and they will still tell a customer asking about you that they found no details.
What comes after structure
Once your website structure is readable, then yes, write a good robots.txt. Use what works for search engines. And if you want to offer AI tools a summary in your own words, publish a plain text file. I call mine llms.txt. It does not have to be long. It is just a way to say: here is what matters, in one place, the way we describe ourselves.
Then set your robots.txt rules: allow the search and answer-engine crawlers, disallow the training bots if you want to (or allow them if you do), and disallow the URLs that waste crawl budget. That is the whole sequence. But the first step is making sure the useful pages are structured so the crawlers can actually read them. That is what determines whether an AI assistant recommends you or leaves your practice out of the answer.