The AI crawler checklist

• Run the free AI crawler checker on your homepage: it tests each AI crawler against your robots.txt and your firewall.
• Open yoursite.com/robots.txt in a browser and read it.

• Search: OAI-SearchBot (ChatGPT), Claude-SearchBot (Claude), PerplexityBot (Perplexity), Googlebot (Google Search, AI Overviews and AI Mode), Bingbot (Bing and Copilot).
• Pages people ask about in a chat: ChatGPT-User, Claude-User, Perplexity-User.

GPTBot, ClaudeBot, CCBot (Common Crawl) and meta-externalagent collect pages to train AI models. Blocking them doesn't take you out of AI answers. Applebot-Extended and Google-Extended are settings rather than crawlers; blocking Google-Extended can stop Gemini using your pages in its answers, so most local businesses leave it allowed.

• Your CDN or firewall (Cloudflare: Security → Bots): allow verified AI search crawlers, even if you block others.
• Your security plugin or hosting: turn off blocks on crawlers it doesn't recognise, or add the AI search crawlers to its allow list.
• No challenge pages (“prove you're human”) for crawlers: they can't pass them.

• View the source of your homepage and main service page, and search for a sentence from each. If it's missing, crawlers that don't run scripts can't see it.
• Important words are text, not pictures of text.
• Pages you want quoted aren't set to noindex, and aren't behind a login.

• A sitemap at yoursite.com/sitemap.xml, submitted in Google Search Console and Bing Webmaster Tools.
• IndexNow, if your website supports it, to tell Bing about new and changed pages straight away.
• Optional: an llms.txt file with a short summary of your business and key pages.

• Run the AI crawler checker again after a redesign, a new plugin, a new host or a firewall change.
• Read robots.txt again every three months.