Cloudflare's AI Bot Settings: What They Block and What to Check

The settings involved

Cloudflare's controls have changed names and defaults over time, so check your dashboard rather than relying on this list. The ones that affect AI crawlers:

AI bot blocking. A single toggle under Security or Bots that blocks known AI crawlers. On some plans and for some newly onboarded zones, it has been on by default. When enabled, requests from GPTBot, ClaudeBot, PerplexityBot, and similar user agents are rejected before they reach your server.

Bot Fight Mode and Super Bot Fight Mode. Broader protection that challenges or blocks traffic it classifies as automated. AI crawlers are automated, and depending on configuration they may be challenged with a JavaScript test they cannot pass.

WAF custom rules. Rules written by your team or an agency. A rule that blocks "all bots except Googlebot" from years ago will block every AI crawler.

Managed robots.txt. Cloudflare can inject directives into your robots.txt that disallow AI crawlers. Your origin file may say allow; the served file may say disallow.

Rate limiting. Aggressive rate limits can cause AI crawlers to receive 429 responses on a fraction of requests, which looks like partial availability.

What blocking actually does

When the AI bot block is on, the crawler receives a 403 and moves on. It does not retry much, and it does not render your page. The engine behind it has no current copy of your site. Depending on the engine, it may fall back to older training data, cite other sites that describe you, or say nothing.

Blocking is also silent from your side. Nothing shows in analytics, because the request never reaches your pages. Nothing shows in Search Console, because Googlebot is unaffected. You find out when an engine describes your company from someone else's page.

Deciding what you want

Some teams block deliberately, to keep content out of training data. That is a legitimate position. The trade-off is that engines that retrieve live pages, which are the ones that cite and link, also cannot reach you.

A middle path is to allow the retrieval crawlers and block the training-only ones. OpenAI, for example, documents separate user agents for training and for search. Whether that distinction holds in practice is uncertain, but the robots.txt and Cloudflare settings can express it.

How to check

Do not trust the dashboard alone. Test from outside.

1. Send a request to a key page with the user agent set to GPTBot, then ClaudeBot, then PerplexityBot. Compare the response code to a normal browser request.
2. Fetch /robots.txt and read what is actually served, not what is in your repository.
3. Check server logs or Cloudflare analytics for 403 and 429 responses filtered by those user agents.
4. If you use a managed challenge, confirm it is not applied to those agents.

A 200 with full HTML is what you want. A 403, a challenge page, or a 200 with an interstitial means you are blocked.

Fixing it

Turn off the AI bot block, or add exceptions for the crawlers you want. Add WAF skip rules for the specific user agents if Super Bot Fight Mode is on. Review custom rules for blanket bot blocks. If Cloudflare manages your robots.txt, check what it adds and override it if needed. Then re-test.

Verify by user agent and, where the crawler publishes IP ranges, by IP, since some rules match on either.

How we handle it

At AEO GEO Labs, crawler access is the first check in every AI visibility audit, before any content work, because there is no point rewriting pages a crawler cannot fetch. Cloudflare settings are the most frequent cause we find, and the fix usually takes less time than the diagnosis did.

Test it today

Run the user agent test above on your homepage and your most important product page. AEO GEO Labs offers a free AI crawler checker that sends those requests for you and reports which bots are blocked, which is the quickest way to find out whether Cloudflare is standing between you and the engines.