Make sure ChatGPT can read your website
If your robots.txt or firewall turns ChatGPT's crawler away, it can't quote you. How to check in a minute, and what to change.
Before anything else, make sure ChatGPT is allowed in. A surprising number of business websites block it without knowing: an old robots.txt line, a security plugin, or a firewall setting that treats AI crawlers as bots to stop.
The crawlers that matter
ChatGPT uses OAI-SearchBot to find pages for its answers and ChatGPT-User when someone asks it to look at a page. GPTBot is different: it collects pages to train future models. You can block GPTBot if you'd rather not be used for training and still be found in ChatGPT's answers, as long as OAI-SearchBot is allowed.
How to check
- Open yoursite.com/robots.txt in a browser. Look for lines that say Disallow under User-agent: OAI-SearchBot, ChatGPT-User or User-agent: * (all crawlers).
- Check your security plugin or firewall (Cloudflare, Wordfence, Sucuri) for a setting that blocks AI bots or “bad bots”. Cloudflare has a one-click AI crawler block that's easy to turn on by accident.
- Make sure your important pages show their text without needing a click or a login, and aren't marked noindex.
Our free AI crawler checker does all of this in about 15 seconds and tells you exactly which line or setting is in the way.
A short summary for AI
Some sites now publish an llms.txt file: a short plain-text summary of who you are, what you do, where, and your key pages. It's quick to add and does no harm. Our free generator writes one from your website.
Do this before the next lesson
- Run the free AI crawler checker on your website.
- If anything is blocked, change that line or setting (or send the result to whoever runs your website), then run it again.
- Optional: make an llms.txt with the free generator and add it to your site.
In Groow
GroowBot checks your website every week, tells you if an AI crawler starts getting turned away, and counts the visits AI crawlers make to your pages.
Try Groow free