BlogWebsite
Is Your Website Blocking AI? A 5-Minute Check for Contractors
By the Groow team · · 5 min read
{
"@type": "LocalBusiness",
"name": "Northbeam Heating & Air",
"areaServed": "Des Moines, IA",
"makesOffer": {
"@type": "Service",
"name": "Furnace replacement"
},
"aggregateRating": { "ratingValue": 4.9 }
}If AI crawlers can't read your website, they can't cite it. Here's how to check your robots.txt and security settings in five minutes.
AI assistants that search the web send crawlers to read pages before they cite them. If your website blocks those crawlers, often by accident, your pages can't show up as a source in their answers, no matter how good they are.
Step 1: Look at your robots.txt
Open your website address followed by /robots.txt, for example yourcompany.com/robots.txt. This file tells crawlers what they're allowed to read. Look for lines that say Disallow next to a crawler's name, or a blanket “User-agent: *” followed by “Disallow: /”, which blocks everyone.
Step 2: Know the crawler names
- OpenAI: OAI-SearchBot is used for ChatGPT search. GPTBot is used for training.
- Perplexity: PerplexityBot
- Anthropic (Claude): ClaudeBot and Claude-SearchBot
- Google: Googlebot powers Search and AI Overviews. Google-Extended controls whether your content is used for Gemini, and blocking it doesn't remove you from Search.
- Microsoft Copilot relies on Bing, whose crawler is Bingbot.
You can choose to block training crawlers and still allow search crawlers. If you want to appear in AI answers, don't block the search ones.
Step 3: Check your hosting and security settings
Some website builders, hosting companies and security services, including Cloudflare, have settings that block AI bots. They can be switched on without you realizing it. Ask whoever manages your website to check.
Step 4: Make your pages easy to understand
- Give each main service its own page, with the city you serve in the heading.
- Answer common questions directly, like cost ranges, financing and response time.
- Add structured data (LocalBusiness, Service and FAQ schema) so engines know exactly what you offer and where.
Not sure where your site stands? The free scan checks how AI engines see your business, including the sources they cite.
