03.03Lesson6 min
Firewalls and plugins that turn AI away
Your robots.txt can say yes while your firewall says no. Where AI crawlers get blocked without anyone deciding to, and what to change.
Robots.txt is a polite request. Firewalls, security plugins and hosting companies can block crawlers outright before robots.txt is even read, and some do it by default.
Where it happens
- Content delivery networks like Cloudflare. Cloudflare has settings to block AI crawlers, and new sites are asked about them when they sign up, so it's easy to have them on without realising. Look under Security, then Bots.
- Security plugins like Wordfence, and bot protection from your hosting company, which may challenge or block crawlers they don't recognise.
- Challenge pages that ask visitors to prove they're human. Crawlers can't, so they get nothing.
What to change
- Allow the verified AI search crawlers in your firewall or CDN, even if you block other bots.
- If you want to block AI training, use the training settings, not a block on every AI crawler.
- If someone else runs your website, send them the crawler checker's result and ask them to check.
After any change to your firewall, hosting or plugins, run the crawler checker again. Website changes undo this more often than anything else.
Do this before the next lesson
- Run the crawler checker and look for crawlers that got a challenge page or an error.
- Check your CDN, firewall and security plugin for AI or bot blocking.
- Allow the AI search crawlers, then run the checker again.
In Groow
Groow tests your site the way each AI crawler sees it every week, and tells you when a firewall or plugin starts turning one away.
Put this to work on your business
Groow checks how ChatGPT, Google and Google Maps see your business every week, tells you what to fix first, and drafts the work for your OK.