ExplainerTechnical AEO
AI Crawlers and robots.txt: Which Bots to Allow If You Want to Be Cited
by the AEO GEO Labs team3 min read
A surprising number of companies that want to appear in AI answers have blocked the crawlers that would put them there. It usually happens in one line of robots.txt, added during a wave of concern about AI training, that also blocks the search crawlers answer engines use to read pages today. Here is how to tell the two apart and set the file deliberately.
Training crawlers versus search crawlers
AI companies run different bots for different purposes, and the distinction matters:
- Training crawlers collect pages to train future models. Blocking them is a legitimate choice about your content's use in training, and it has little effect on whether you appear in answers today.
- Search and retrieval crawlers fetch pages at query time so an assistant can read and cite them. Blocking these removes you from answers that use live search.
Several providers publish separate user agents for each purpose. The practical rule: if you want to be cited, the search crawlers must be allowed, whatever you decide about training.
The user agents to know
Published user agents as of this writing, which you should verify against each provider's current documentation since names change:
- OpenAI:
GPTBot(training),OAI-SearchBot(search and citations in ChatGPT), andChatGPT-User(fetches on behalf of a user's request). - Anthropic:
ClaudeBotand related agents; Anthropic documents its crawler behavior and respects robots.txt. - Perplexity:
PerplexityBotfor its index, with a separate user-triggered fetch agent. - Google:
Googlebotserves Search, including AI Overviews.Google-Extendedis a control for whether content is used for Gemini training and grounding, and blocking it does not remove you from Search or Overviews. - Microsoft:
Bingbotserves Bing and Copilot.
The common mistake is a rule like User-agent: GPTBot / Disallow: / copied from a template that also includes OAI-SearchBot or, worse, a wildcard that catches everything.
A deliberate robots.txt
If your goal is to be cited while declining training use, the shape is:
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: *
Allow: /
If you are comfortable with training use, simply do not block the training agents. Either way, decide on purpose, and document the decision so the next person editing the file knows why each line exists.
Beyond robots.txt
Blocking happens elsewhere too:
- CDN and WAF bot rules. Cloudflare and similar services offer one-click "block AI bots" settings that block search crawlers as well. Check the setting, not just the file.
- Rate limiting. Aggressive limits can cause crawlers to give up on deep pages.
- JavaScript-only rendering. Not a block, but the same outcome: the crawler fetches the page and finds nothing to read.
- Login walls and interstitials. Cookie banners that gate content, geo-blocks, and age gates can all hide pages from crawlers.
How to check
Fetch your key pages with each crawler's user agent string and confirm you get the full HTML with the content present. Check server logs for the agents above and see whether they are being served 200s or 403s. Review the CDN bot-management settings. AEO GEO Labs offers a free AI crawler checker that runs the first of these for you, and the audit always includes the full check, because a crawl block makes every other visibility effort pointless.
Do this now
Open robots.txt and look for any Disallow applied to a search crawler. Then open your CDN's bot settings. Ten minutes of checking has, more than once, explained a company's total absence from AI answers. If you want the full crawl and rendering check across every engine's agents, AEO GEO Labs runs it as part of the free report.