Robots.txt: the rules crawlers follow
A short file on your site tells crawlers where they may go. How to read it, the mistakes that block AI by accident, and how to opt out of training only.
Most websites have a robots.txt file at yoursite.com/robots.txt. Well-behaved crawlers read it first and follow its rules. It's plain text, and you can open yours in a browser right now.
How to read it
- User-agent names the crawler a rule is for. User-agent: * means every crawler.
- Disallow names what that crawler may not read. Disallow: / means the whole site.
- Allow names what it may read, as an exception.
The mistakes that block AI by accident
- A Disallow: / under User-agent: * left over from when the site was being built.
- A plugin or template that blocks every crawler it doesn't recognise.
- Lines copied from a list of “bad bots” that includes OAI-SearchBot, Claude-SearchBot or PerplexityBot.
Opting out of training, staying in the answers
You can block the training crawlers and still be found in AI answers, as long as the search crawlers stay allowed. For example:
A robots.txt that blocks training but not search
User-agent: GPTBot Disallow: / User-agent: CCBot Disallow: / User-agent: * Allow: /
Be careful with Google-Extended. Blocking it keeps you in Google Search, AI Overviews and AI Mode, but Gemini may stop using your pages in its own answers. Most local businesses are better off leaving it allowed.
Do this before the next lesson
- Open yoursite.com/robots.txt and read it with the three rules above.
- Remove anything that blocks OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot, Googlebot or Bingbot.
- Decide whether to block the training crawlers, and add those lines if so.
In Groow
Groow reads your robots.txt every week and tells you if a change starts blocking an AI search crawler.
Put this to work on your business
Groow checks how ChatGPT, Google and Google Maps see your business every week, tells you what to fix first, and drafts the work for your OK.