Sitemaps, llms.txt, and keeping it fixed
Help crawlers find every page with a sitemap, add an llms.txt if you like, and check again after every change to your website.
A sitemap
A sitemap is a list of your pages, usually at yoursite.com/sitemap.xml. Most website builders make one for you. Submit it in Google Search Console and Bing Webmaster Tools so both find new pages quickly. Bing also supports IndexNow, which tells it about new and changed pages straight away, and Bing's index feeds Microsoft Copilot.
llms.txt
An llms.txt file is a short plain-text summary of your business and your key pages, written for AI. It's a newer idea and few assistants have said they use it, so don't expect it to change much on its own. It's quick to add and does no harm, and the free generator writes one from your website.
Visits from AI assistants
Google Analytics · last 8 weeks
41 visits last week, up from 18 eight weeks ago
- ChatGPT58%
- Perplexity21%
- Gemini12%
- Copilot9%
Sample data
Keep it fixed
- Run the crawler checker after every website change: a redesign, a new plugin, a new host or a firewall change.
- Read your robots.txt again every three months.
- Look for AI crawler visits in your website stats, if your host shows them.
Do this before the next lesson
- Find your sitemap and submit it in Google Search Console and Bing Webmaster Tools.
- Optional: make an llms.txt with the free generator and add it to your site.
- Set a reminder to run the crawler checker after any website change.
In Groow
Groow checks all of this every week, tells Bing about your new and changed pages through IndexNow, and counts the visits AI crawlers make to your website.
Put this to work on your business
Groow checks how ChatGPT, Google and Google Maps see your business every week, tells you what to fix first, and drafts the work for your OK.