03.01Lesson5 min

How AI reads your website

AI assistants send crawlers to read web pages. The three kinds, which ones matter for being found, and what they can and can't see.

Before an AI assistant can quote your website, its crawler has to read it: a program that fetches your pages the way a browser does and takes the text. If the crawler is turned away, the assistant can only repeat what other websites say about you, if anything.

Three kinds of crawler

  • Search crawlers find and read pages for answers: OAI-SearchBot for ChatGPT, Claude-SearchBot for Claude, PerplexityBot for Perplexity, and Googlebot and Bingbot, which feed Google's AI Overviews and AI Mode and Microsoft Copilot.
  • Chat fetchers open a page when someone asks about it in a conversation: ChatGPT-User, Claude-User and Perplexity-User.
  • Training crawlers collect pages to train future AI models: GPTBot, ClaudeBot, CCBot (Common Crawl) and Meta's crawler. Google-Extended and Applebot-Extended are settings rather than crawlers: they say whether Google and Apple may use your pages for their AI.

The first two kinds decide whether you can be found and quoted. The third is a choice about training, and the next lesson covers it.

Engines tracked

Weekly

  • ChatGPT

    ChatGPT

    64%

  • Claude

    Claude

    33%

  • Gemini

    Gemini

    39%

  • Perplexity

    Perplexity

    71%

  • Google AI

    Google AI

    48%

  • Copilot

    Copilot

    Soon

  • Grok

    Grok

    Soon

  • Meta AI

    Meta AI

    Soon

  • DeepSeek

    DeepSeek

    Soon

Sample data

Each assistant reads the web with its own crawler.

What crawlers see

Crawlers read the text in your page's code. Most AI crawlers don't run JavaScript, so text that only appears after scripts run, words inside images, and anything behind a login can be invisible to them. Lesson four shows how to check.

Do this before the next lesson

  • Run the free AI crawler checker on your website.
  • Note any crawler marked as blocked, and whether it's a search crawler or a training one.
Run the AI crawler checker

In Groow

GroowBot checks your website every week, tells you if an AI crawler starts getting turned away, and counts the visits AI crawlers make to your pages.

Put this to work on your business

Groow checks how ChatGPT, Google and Google Maps see your business every week, tells you what to fix first, and drafts the work for your OK.

Get my free AI scanStart free, no card