Robots.txt tester: which line decides?

robots.txt looks simple, and most mistakes come from assuming it's read top to bottom. It isn't. Since RFC 9309 (2022), and in Google's parser for years before that, a crawler does this:

1. 1Find its group

   The group whose User-agent names the crawler wins. Only if there's none does it fall back to User-agent: *. It never uses both.

2. 2Collect matching rules

   Every Allow and Disallow in that group whose path matches the start of the URL, with * and $ wildcards.

3. 3Longest wins

   The most specific (longest) matching rule decides. If an Allow and a Disallow tie, Allow wins.

4. 4No match means allowed

   An empty Disallow, or no matching rule, allows the URL. /robots.txt itself is always allowed.

The tester implements exactly that, keeps line numbers for every rule, and shows the group a crawler ends up in. It also flags lines crawlers will skip: unknown directives, rules before any User-agent, paths that don't start with a slash, and Noindex, which Google stopped supporting in robots.txt in 2019.

A named group that forgets the basics. Adding User-agent: Googlebot with one rule means Googlebot no longer sees your User-agent: * exclusions. Repeat the rules you still want in the named group.

Blocking CSS and JavaScript. Disallowing /assets/ or /wp-includes/ stops Google from rendering your pages properly. The tester warns when a rule looks like it covers asset files.

Blanket AI blocks. A Disallow: / under a list of AI user agents often catches search crawlers like OAI-SearchBot and PerplexityBot along with the training ones. Check what each AI crawler is allowed with the AI crawler checker.

A robots.txt that errors. If /robots.txt returns a 5xx, Google treats the whole site as disallowed until it can read the file again. A 404 is fine; it means “no restrictions”.

Pick a starting point, adjust individual crawlers, list the paths you want kept out of every crawler, and add your sitemap. The presets reflect the real choices: allow everything, let AI search cite you while opting out of model training, or block AI entirely and keep classic search.

Your draft is saved in this browser. Paste the output into the tester tab to check it against your real URLs before you upload it, and point a sitemap validator at the sitemap you list.