BlogWebsite
Is Your Website Blocking AI? A 5-Minute Check for Contractors
Kyle Vogt, Founder, Groow GroupPublished · 5 min read
{
"@type": "LocalBusiness",
"name": "Northbeam Heating & Air",
"areaServed": "Des Moines, IA",
"makesOffer": {
"@type": "Service",
"name": "Furnace replacement"
},
"aggregateRating": { "ratingValue": 4.9 }
}If AI crawlers can't read your website, they can't cite it. Here's how to check your robots.txt and security settings in five minutes.
On this page
Key takeaways
- If AI crawlers can't read your website, they can't cite it, however good the page is.
- Check your robots.txt for blocks on search crawlers like OAI-SearchBot, PerplexityBot, Claude-SearchBot and Googlebot.
- You can block training crawlers and still allow the search ones.
- Hosting and security settings can block AI bots without you knowing, so ask whoever manages your site to check.
AI assistants that search the web send crawlers to read pages before they cite them. If your website blocks those crawlers, often by accident, your pages can't show up as a source in their answers, no matter how good they are.
Step 1: Look at your robots.txt
Open your website address followed by /robots.txt, for example yourcompany.com/robots.txt. This file tells crawlers what they're allowed to read. Look for lines that say Disallow next to a crawler's name, or a blanket “User-agent: *” followed by “Disallow: /”, which blocks everyone.
Step 2: Know the crawler names
- OpenAI: OAI-SearchBot is used for ChatGPT search. GPTBot is used for training.
- Perplexity: PerplexityBot
- Anthropic (Claude): ClaudeBot and Claude-SearchBot
- Google: Googlebot powers Search and AI Overviews. Google-Extended controls whether your content is used for Gemini, and blocking it doesn't remove you from Search.
- Microsoft Copilot relies on Bing, whose crawler is Bingbot.
You can choose to block training crawlers and still allow search crawlers. If you want to appear in AI answers, don't block the search ones.
Step 3: Check your hosting and security settings
Some website builders, hosting companies and security services, including Cloudflare, have settings that block AI bots. They can be switched on without you realizing it. Ask whoever manages your website to check.
Step 4: Make your pages easy to understand
- Give each main service its own page, with the city you serve in the heading.
- Answer common questions directly, like cost ranges, financing and response time.
- Add structured data (LocalBusiness, Service and FAQ schema) so engines know exactly what you offer and where.
Not sure where your site stands? The free scan checks how AI engines see your business, including the sources they cite.
About the author
Kyle Vogt, Founder, Groow Group
Kyle founded Groow Group to help businesses show up when customers ask ChatGPT, Google and the other AI assistants who to call. He writes about what actually moves AI and Google visibility, from the checks Groow runs every week.