Free AI crawler checker
Can ChatGPT, Claude, Perplexity, Google and Bing read your website? Enter it and we'll check your robots.txt for every AI crawler, from OAI-SearchBot and ClaudeBot to GPTBot and PerplexityBot, then knock as each one to see whether your firewall lets it in.
Updated
Free, no sign-up. Takes about 15 seconds.
The AI crawlers, and what each one does
AI companies send different crawlers for different jobs. Some read pages so their assistant can answer questions and cite you: these decide whether you can show up in AI answers. Some collect pages to train future models. And some open a page only when a person asks the assistant to.
OpenAI (ChatGPT)
- OAI-SearchBot
- Builds the index ChatGPT search uses to find and cite websites. Keep it allowed.
- ChatGPT-User
- Opens a page when someone asks ChatGPT to, or when it looks something up for them.
- GPTBot
- Collects public pages to train OpenAI's models. Blocking it doesn't remove you from ChatGPT search.
Anthropic (Claude)
- Claude-SearchBot
- Indexes pages so Claude can find and cite them when it searches the web. Keep it allowed.
- Claude-User
- Opens a page when someone asks Claude to.
- ClaudeBot
- Collects public pages to train Claude models. Blocking it doesn't stop Claude from finding you in search.
Perplexity
- PerplexityBot
- Indexes pages for Perplexity's answers. Keep it allowed.
- Perplexity-User
- Opens a page when someone asks Perplexity to.
- Googlebot
- Google Search, which AI Overviews and AI Mode are built on. Blocking it removes you from all three.
- Google-Extended
- Not a separate crawler: a robots.txt setting for whether Google may use your pages to train and ground Gemini. It doesn't change Google Search or AI Overviews.
Microsoft
- Bingbot
- Bing search, which Microsoft Copilot and ChatGPT search lean on. Keep it allowed.
Others
- Applebot-Extended
- A robots.txt setting for whether Apple may use your pages to train its AI models.
- CCBot
- Common Crawl, a free public archive of the web that many AI models are trained on.
- meta-externalagent
- Collects pages for Meta's AI models and products.
What is ClaudeBot?
ClaudeBot is Anthropic's web crawler. It collects publicly available pages that may be used to train Claude models, so if you see ClaudeBot in your server logs, that's what it's doing. It follows robots.txt, so a ClaudeBot rule in yours decides whether it reads your site.
Blocking ClaudeBot doesn't stop Claude from finding your site when it searches the web. That's Claude-SearchBot, plus Claude-User when a person asks Claude to open a page. If you want Claude to be able to recommend you, keep those two allowed, whatever you decide about ClaudeBot.
What is GPTBot, and what is OAI-SearchBot?
OpenAI runs the same split. GPTBot collects pages to train OpenAI's models; OAI-SearchBot builds the index ChatGPT search uses to find and cite websites; ChatGPT-User opens pages when someone asks ChatGPT to. Blocking GPTBot keeps your pages out of future training, but only blocking OAI-SearchBot keeps you out of ChatGPT search.
Should you block AI crawlers?
- Search and assistant crawlers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, Bingbot): keep them allowed if you want customers to find you through AI. Block them and those assistants can't read, recommend or cite you.
- Training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, meta-externalagent): your call. Some businesses allow them so future models know about them; publishers often block them to keep their writing out. Either way, it doesn't change today's AI answers.
How to allow or block AI crawlers in robots.txt
robots.txt is a plain text file at yourwebsite.com/robots.txt. Each group names one or more crawlers with User-agent lines, then says what they may read. A crawler follows the group that names it, and only falls back to the User-agent: * group when none does. This example lets the AI search crawlers in and keeps three training crawlers out:
# Let AI search crawlers in User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot Allow: / # Keep training crawlers out (optional) User-agent: GPTBot User-agent: ClaudeBot User-agent: CCBot Disallow: /
Why a crawler can be blocked even when robots.txt allows it
robots.txt is only half of it. Some hosts and firewalls, Cloudflare among them, have settings that block AI crawlers, and they can be on without you knowing. The checker knocks on your home page as each crawler and compares what comes back with what a normal browser gets. A firewall may treat our test as an impostor and still let the real crawler in, so a block here means check the setting, not proof.
Questions
Is ClaudeBot safe to allow?
It's Anthropic's crawler and it follows robots.txt. Allowing it means your public pages may be used to train Claude models. Blocking it is fine too: it doesn't stop Claude from finding your site when it searches the web.
Does blocking GPTBot keep me out of ChatGPT?
No. GPTBot is OpenAI's training crawler. ChatGPT search finds and cites websites through OAI-SearchBot, and opens pages for people through ChatGPT-User. Keep those two allowed if you want ChatGPT to be able to recommend you.
Does robots.txt actually stop AI crawlers?
The big AI companies say their crawlers follow it. It's a request, not a lock: a crawler that ignores it can only be stopped by a firewall.
Is this a full robots.txt tester?
It checks your robots.txt for the AI crawlers and search engines that matter for AI answers, on your home page. For testing other crawlers or specific pages, Google Search Console and Bing Webmaster Tools have their own robots.txt reports.
What does the checker do with my website?
It reads your robots.txt and your home page, once as a normal browser and once as each AI crawler: about 20 polite requests. We save the result so the same site isn't checked over and over (repeat checks within an hour get the saved one), and a scrambled code made from your network address to limit how often it runs, never the address itself. No sign-up, no email.
Read next
Let's get the work done.Talk to a founder, or start free. No pitch.
Free plan, no card · From $149 a month · Cancel anytime