XML Sitemaps and AI Crawlers: Small Details That Decide Discovery

What AI crawlers do with sitemaps

Traditional search crawlers discover most pages through links and use sitemaps as a supplement. Newer AI crawlers often have less history with your site and less patience. A clean sitemap gives them a complete, prioritized list without needing to crawl your navigation. Some also appear to read lastmod to decide what to re-fetch, which makes that field more valuable than it was.

None of this is guaranteed. No AI crawler publishes a full description of its discovery logic. But a sitemap costs nothing to get right, and getting it wrong removes one of the few levers you have.

The details that matter

It must be reachable. Check that /sitemap.xml returns 200 and that robots.txt includes a Sitemap: line pointing to it. Then check that the AI crawlers themselves are not blocked from fetching it. A sitemap behind a firewall rule that blocks GPTBot is useless to GPTBot.

It must contain only canonical, indexable URLs. Every URL should be the canonical version, return 200, and not carry a noindex tag. A sitemap full of redirects, 404s, or noindexed pages wastes crawl budget and signals carelessness.

lastmod must be honest. Set it from the real content change date, not the build time. A sitemap where every URL was "modified" today, every day, tells the crawler nothing. One where lastmod is accurate lets it prioritize what actually changed.

It must be current. New pages should appear within hours, not weeks. Deleted pages should disappear. If your sitemap is a static file someone regenerates by hand, it is probably wrong right now.

It should be split sensibly. Sitemaps cap at 50,000 URLs and 50 MB. Larger sites use a sitemap index pointing to multiple files. Splitting by content type (pages, posts, products, docs) makes it easier to see which section is stale.

Common mistakes

• Listing paginated, filtered, or parameterized URLs that duplicate canonical pages.
• Including staging or development URLs that leaked into the build.
• Omitting the pages that matter most because they live in a separate app or subdomain with its own sitemap that nobody linked from robots.txt.
• Setting priority and changefreq and expecting them to matter. Most crawlers ignore both.
• Serving the sitemap with the wrong content type or gzip errors so it fails to parse.

Sitemaps and JavaScript sites

Single-page applications often generate sitemaps from a route list rather than from rendered pages. That can include routes that render nothing without client-side data, which the crawler fetches and finds empty. Make sure each URL in the sitemap serves real content in the initial HTML response, or at least a server-rendered version.

Checking what the crawlers see

Pull your server logs and filter for the AI user agents: GPTBot, ClaudeBot, PerplexityBot, and the others you care about. Look at which URLs they request and whether the sitemap is among them. If they fetch the sitemap and then request the URLs in it, the channel is working. If they fetch the sitemap and nothing else, something in the sitemap is wrong, or the pages themselves are blocked.

Compare the list of pages you want cited with the list the crawlers actually fetched. The gap is your to-do list.

How we approach it

At AEO GEO Labs, the sitemap check is part of the technical layer of every AI visibility audit, alongside robots.txt, crawler access, and rendering. It is quick, and it frequently turns up a reason why a well-written page is invisible: it was never in the sitemap, or it was listed under a URL that redirects.

Five-minute check

Open your sitemap, pick ten URLs at random, and visit each one. Confirm they return 200, are canonical, and are not noindexed. Then check that the crawler user agents can fetch the sitemap itself. AEO GEO Labs offers a free AI crawler checker that tests whether the main AI bots can reach your pages, which covers the access half of this in one pass.