BlogHow-to
Is ChatGPT Allowed to Read Your Website? How to Check in Two Minutes
By the Groow team · · 5 min read
GEO & SEO audit · stackwise.io
1,200 pages
- OAI-SearchBot (ChatGPT)AllowedOK
- PerplexityBotBlocked in robots.txtUrgent
- ClaudeBotAllowedOK
- FAQ / Product schemaMissing on 12 key pagesFix
- Titles and descriptions38 duplicatesFix
- Comparison pagesNot linked from navigationFix
Some websites block AI assistants without knowing it. How to check your robots.txt, which AI bots matter for being recommended, and what to change.
If an AI assistant can't read your website, it can't quote it, and it's much less likely to recommend you. Some websites block AI assistants without the owner ever deciding to: a security plugin, a hosting setting or a template copied years ago can do it quietly.
The good news is that it takes about two minutes to check.
Step 1: Open your robots.txt
Type your website address followed by /robots.txt into your browser, like yoursite.com/robots.txt. This short text file tells bots which parts of your site they may read. If the page is empty or missing, nothing is blocked there.
Step 2: Look for these names
Each AI company uses its own bots. The ones that matter most for being recommended are the ones that read pages to answer people's questions:
- OAI-SearchBot and ChatGPT-User (ChatGPT)
- Claude-SearchBot and Claude-User (Claude)
- PerplexityBot and Perplexity-User (Perplexity)
- Googlebot (Google Search, including AI Overviews and AI Mode)
- Bingbot (Bing and Microsoft Copilot)
A block looks like a User-agent line with the bot's name, followed by “Disallow: /”. Also check for “User-agent: *” followed by “Disallow: /”, which blocks every bot from the whole site.
Training bots are a separate choice
Some bots collect pages to train future AI models rather than to answer questions today: GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Gemini) and CCBot (Common Crawl), among others. Blocking these is a reasonable business decision, and it doesn't remove you from AI answers. Blocking the search bots above does.
Rule of thumb: let the search bots in, and decide on the training bots for yourself.
Step 3: Check your firewall and plugins
Even when robots.txt allows a bot, a firewall can still turn it away. Some hosting providers, CDNs and security plugins have a one-click setting to block AI bots. If you use one, look for that setting and make sure it isn't blocking the search bots you want.
Step 4: Fix it
If a search bot is blocked, remove its Disallow line, or change “Disallow: /” to “Allow: /” for that bot. If a plugin or host wrote the file, change the setting there instead, or your edit may be overwritten. Whoever manages your website can usually do this in a few minutes.
How Groow checks this for you
GroowBot, Groow's crawler, checks every week whether ChatGPT, Claude, Perplexity, Google and Bing can read your site, reading robots.txt the way each bot does and asking for your home page as each one. If something changes, it shows up in your site audit and your recommendations, with what to fix.
The free AI scan runs the same check on your website in about a minute.
