How-toChatGPT, Gemini & co.
Claude and AI Assistants: Do They Browse, and What Do They Cite?
by the AEO GEO Labs team5 min read
AI assistant citations are the source links an assistant like Claude, ChatGPT, Gemini or Perplexity shows when it answers using the live web. As of October 2026, all four can browse, but not on every question: they search when a question needs current or specific facts, and answer from training data otherwise. To be cited, your page must be reachable by the assistant's search bot and answer the question directly.
This guide is part of our LLM SEO guide. It focuses on Claude, then compares the other assistants.
Do AI assistants browse the web?
Yes, but selectively. An assistant answers from what it learned in training unless the question needs fresher or more specific information, in which case it runs a web search and cites what it reads.
Anthropic's web search tool documentation spells this out for Claude. Claude searches for recent events, current prices and statistics, and information about specific organizations, people or products that may have changed. It answers directly, without searching, for stable knowledge such as established facts, math and coding concepts.
That split matters for site owners. Questions about your product, pricing, comparisons and recent changes are exactly the ones that trigger a search. Those are the citations you can win.
How Claude finds and cites sources
Claude finds sources through a web search tool, then cites the passages it uses. Anthropic launched web search in Claude in March 2025 and, per its announcement, made it available globally on all Claude plans in May 2025.
Three details from Anthropic's documentation, as of October 2026:
- Citations are always on for web search in the API. Each citation includes the URL, the page title, and up to 150 characters of the cited text.
- Claude may search several times in one answer. Anthropic's docs say simple factual queries typically use one to three searches, and comparative research can use ten or more.
- Developers can restrict domains. Apps built on Claude can allow or block specific domains, so a third-party product may only cite from a set list of sites.
Anthropic runs three bots, described in its crawler help article:
| Bot | What it does | If you block it |
|---|---|---|
| ClaudeBot | Collects content that may be used for model training | Future content is excluded from training data |
| Claude-SearchBot | Indexes content to improve search results | Your visibility in Claude's search results may drop |
| Claude-User | Fetches a page when a user's question needs it | Claude can't retrieve your page for that user |
Anthropic says all its bots honor robots.txt. That differs from some competitors, as the next section shows.
AI assistant citations compared: Claude, ChatGPT, Perplexity, Gemini
Each assistant uses a different search stack and different bots. This table summarizes what each company documents as of October 2026.
| Assistant | Search source | Bot that builds search visibility | User-triggered fetcher and robots.txt |
|---|---|---|---|
| Claude | Anthropic's web search tool | Claude-SearchBot | Claude-User, honors robots.txt |
| ChatGPT | OpenAI's crawl plus third-party providers | OAI-SearchBot | ChatGPT-User, "robots.txt rules may not apply" |
| Perplexity | Perplexity's own index | PerplexityBot | Perplexity-User, "generally ignores robots.txt" |
| Gemini | Google Search grounding | Googlebot | Google's standard controls |
Sources: OpenAI's crawler documentation, Perplexity's bot documentation, and Google's Grounding with Google Search docs.
A common mistake: blocking every AI bot in one robots.txt rule. That removes you from assistant search results, not only from training. Block training bots like ClaudeBot or GPTBot separately if you prefer, and keep the search bots allowed.
How to get cited by Claude and other assistants
The steps below work across assistants, because they all share the same search-then-cite pattern.
- Allow the search bots. Permit Claude-SearchBot, Claude-User, OAI-SearchBot, PerplexityBot, Bingbot and Googlebot. Our AI Crawler Checker tests your robots.txt against each.
- Don't block by IP. Anthropic notes that IP blocking can stop its bots from reading your robots.txt, so your stated preferences can't be applied.
- Serve key facts in HTML. User-triggered fetchers read a page quickly; text that appears only after JavaScript runs may be missed.
- Write for the questions that trigger search. Pricing, comparisons, setup steps, limits, and "what changed" pages match the kinds of questions where assistants search.
- Make quotable spans. With citations capped at short excerpts, a 1 to 2 sentence answer that names the subject is the unit you want to own.
- Keep facts current and dated. Claude's search results include a page age field, and assistants search precisely because facts change.
For the wider playbook, see how to earn AI citations, and for how engines rank candidates, how AI search engines choose their sources.
Frequently asked questions
Does Claude search the web?
Yes. As of October 2026, Claude can search the web on all plans, a feature Anthropic launched in March 2025 and expanded globally in May 2025. Claude decides when to search, typically for current events, prices, statistics, or details about specific products and organizations. When it uses web content, it cites the sources it drew on.
Which bot should I allow for Claude to cite my site?
Allow Claude-SearchBot and Claude-User. Claude-SearchBot indexes content for Claude's search results, and Claude-User fetches pages when a user's question needs them. Anthropic says blocking either can reduce your visibility. You can block ClaudeBot, the training crawler, separately without affecting the other two.
Why didn't the AI assistant cite any source for my question?
The assistant probably answered from its training data without searching. Assistants generally skip the web for stable knowledge and search for current or specific facts. Asking about recent changes, prices or a specific product usually triggers a search. In ChatGPT and Claude you can also ask it to search the web explicitly.
Do AI assistants cite the same sources as Google?
Often not. Each assistant uses its own search stack: Claude uses Anthropic's web search tool, ChatGPT uses OpenAI's crawl plus other providers, Perplexity runs its own index, and Gemini grounds in Google Search. The same question can produce different citations in each, so check each assistant separately.
If you want this done for your site, AEO GEO Labs runs GEO, AEO and SEO programs for B2B and SaaS teams. See our services.