Crawl Budget for AI Bots: Why Large Sites Get Partially Read

The practical consequence: a large site often has its homepage, a few top-level pages, and a random sample of everything else in an AI engine's index. The pages you most want cited may never have been fetched.

Why AI crawlers fetch less

A few reasons stack up.

AI crawlers are younger and run on less infrastructure than a search crawler. They tend to prioritize breadth across many sites over depth on one site.

They respond badly to slow pages. A page that takes several seconds to respond is a page the crawler may skip and not return to for a long time.

Many of them do not execute JavaScript, or do so inconsistently. A site where navigation and internal links only appear after a script runs gives the crawler almost nothing to follow.

And some crawlers appear to lean on sitemaps and external links to discover URLs rather than deep internal crawling. A page that is four clicks from the homepage and not in a sitemap is easy to miss.

Signs you are partially read

• An engine describes your company accurately but cannot answer questions your documentation covers in detail.
• Your server logs show AI user agents hitting the homepage and a handful of pages repeatedly, and rarely anything deeper.
• Questions about specific products or features are answered from third-party sites instead of your own pages.
• Old URLs that redirect are still being requested, which means the crawler is working from a stale list.

Server logs are the ground truth. Filter by user agent and count unique URLs per bot per week.

What to do

Decide which pages matter. Not all hundred thousand. The pages that answer buyer questions: product pages, pricing, comparisons, documentation entry points, the top support articles. Usually a few hundred at most. Those are the pages the crawl budget should go to.

Put them in a dedicated sitemap. A small sitemap of priority URLs, referenced from robots.txt, is the cheapest way to tell a crawler where to spend its visits. Keep it current and keep it short.

Link them from the pages that do get crawled. The homepage and top-level pages are fetched reliably. Plain HTML links from those pages to the priority pages give a crawler a two-click path. Links that only exist in a JavaScript-rendered mega menu do not count.

Fix response time. Priority pages should respond fast, ideally with a cached HTML response. Anything that routinely takes over a couple of seconds is a page a bot will learn to avoid.

Stop wasting fetches. Faceted navigation, calendar pages, and session parameters burn budget on URLs nobody will ever cite. Block them in robots.txt for the AI user agents.

Check that the priority pages are allowed. Old robots.txt rules often block an entire directory from every bot.

llms.txt as a shortlist

A well-written llms.txt is, in effect, a curated crawl list. Adoption varies by engine, but for a site with a crawl budget problem it costs nothing and gives any crawler that reads it a direct path to the right pages.

Measuring the change

After the sitemap and linking changes, watch the logs. The number of unique priority URLs fetched per bot per week should rise within a few weeks. Then check whether answers to product-specific questions start citing your pages instead of third parties. Share of answer on those questions is the number that proves the budget is being spent well.

Our approach

At AEO GEO Labs, a crawl budget review starts with logs, not assumptions. We map which AI user agents visit, how often, and which URLs they reach, then compare that to the list of pages that should be cited. The gap is the work. On large sites, the fixes are usually unglamorous: a sitemap, a few hundred internal links, and a robots.txt cleanup.

A quick first step

Take your ten most important URLs and run them through the free AI crawler checker at AEO GEO Labs. It reports whether each AI bot is allowed and whether the page is reachable without JavaScript. Then open your server logs and see whether those ten have actually been fetched in the last month. If they have not, you have found the problem.