A large share of the top news publishers online have a mistake sitting in their robots.txt file right now. They meant to block AI companies from scraping their content to train new models. Instead, they also blocked the exact AI tools that send them real visitors. Nobody told them. Their site just quietly stopped showing up when someone asked ChatGPT or Perplexity a question.
Small business sites have the same file, and almost nobody has ever looked at it.
Two Very Different Kinds of AI Bots
Every major AI company now runs two separate bots, and they do very different jobs.
Training bots, like GPTBot and ClaudeBot, crawl your site to build the data a model learns from. Your content becomes part of the model itself. You get no citation, no link back, and no visitor. Anthropic’s own training bots, at one point, crawled roughly 70,900 pages for every single visitor they sent back. That is not a typo. That is the trade a training bot offers, which is close to nothing.
Search bots, like OAI-SearchBot and Claude-SearchBot, work differently. They crawl your site so the AI tool can answer a real question in real time, and when it does, it cites you and links back. That is closer to how a search engine has always worked. It sends you a visitor who is already interested.
Blocking the first kind costs you almost nothing. Blocking the second kind removes you from a growing part of how people find local businesses.
Why the Mistake Happens
Most robots.txt files were never touched after a hosting company or a plugin set a default years ago. When AI bots became a talking point, a lot of site owners added a blanket rule to keep “AI” out, without knowing there are separate bots for separate purposes. Training crawling now makes up roughly 82 percent of all AI bot activity, and search crawling is only around 15 percent. A rule written to stop the first kind can easily catch the second kind too if it is not written carefully.
The safer default for 2026 is simple: block the training bots, allow the search bots. You keep control over how your content gets used to train a model, without losing the visitors an AI search bot would have sent you.
How to Check Your Own Site
Your robots.txt file lives at your domain followed by /robots.txt. Look for any line that disallows all bots, or that specifically names OAI-SearchBot, Claude-SearchBot, or PerplexityBot under a disallow rule. If you see it, that is likely costing you visibility you do not know you are missing.
This is one of the checks in the scanner I built and run against my own sites first. If you want a full picture instead of checking one file by hand, that is exactly what AI agent readiness for Bradenton businesses covers, alongside your llms.txt file and structured data. I wrote about the bigger picture of how AI decides who to recommend in how AI chooses which business to recommend.
If you are not sure what your own robots.txt file says right now, that is worth a quick look before you assume your site is fine. You can reach out through my contact page if you want a second set of eyes on it.