How does robots.txt decide which rule applies?

By specificity, not by order. Every rule in the applicable group is tested against the path, and the one with the longest matching path wins, measured with wildcards removed. If an Allow and a Disallow match at exactly the same length, Allow wins. This is why Allow: /wp-admin/admin-ajax.php overrides Disallow: /wp-admin/ regardless of which line comes first.

Do rules from User-agent: * apply on top of a named group?

No, and this is the most common misunderstanding. A crawler uses exactly one group: the most specific one that names it, and if none names it, the catch-all group. Rules do not stack. If you create a group for GPTBot containing one Disallow line, GPTBot ignores everything in your User-agent: * group, including any Sitemap-adjacent rules you assumed were global.

What is the difference between Disallow and noindex?

Disallow stops a crawler fetching a URL. Noindex tells it not to keep the URL in the index. They are different mechanisms and blocking is not hiding: a disallowed URL can still be indexed and displayed without a snippet if other pages link to it. To remove a page from search, allow crawling and serve a noindex meta tag or X-Robots-Tag header.

Does Noindex: work in robots.txt?

No. Google stopped supporting an unofficial noindex directive in robots.txt in 2019, and lines like that are now silently ignored. Any page you believe is hidden by one is not hidden at all. This tool flags those lines because they are still surprisingly common in files generated by older plugins.

Should I block AI crawlers in robots.txt?

It depends on your business model, and the distinction that matters is training versus retrieval. Training crawlers such as GPTBot, CCBot, Google-Extended, and Applebot-Extended feed model training. Retrieval crawlers such as OAI-SearchBot and PerplexityBot fetch pages to answer a question now, and they are what produce citations and referral traffic. Publishers often restrict the first group and allow the second. Most other businesses lose more from invisibility than they gain from withholding.

Why are my pages still not indexed when robots.txt allows them?

Because robots.txt only controls fetching. Crawlable pages still have to be renderable, canonical to themselves, worth indexing in Google's judgement, and reachable through internal links. A clean robots.txt is a precondition rather than a cause, so if this tool says allowed and the page is still missing, the problem is elsewhere.

Does robots.txt block bad bots or scrapers?

No. It is a set of requests that well-behaved crawlers choose to honour, and it is publicly readable by anyone. Listing your admin paths in it tells a hostile visitor exactly where to look. Anything that genuinely needs protecting needs authentication, rate limiting, or firewall rules, not a text file asking politely.

What happens if my robots.txt returns an error?

It matters more than most people expect. A 404 or 410 is treated as no restrictions, so everything is crawlable. A 5xx server error is treated as a full disallow for a period, meaning a robots.txt outage can stop crawling of your entire site. Serving 503 on your robots.txt during a deploy is a genuinely damaging accident.

Are wildcards supported?

Yes, by Google, Bing, and most major crawlers, though they are extensions rather than part of the original standard. An asterisk matches any sequence of characters and a dollar sign anchors the end of the URL, so Disallow: /*.pdf$ blocks PDFs. Not every crawler implements them, so avoid depending on wildcards for anything critical.

Is there a size limit on robots.txt?

Google reads the first 500KiB and ignores everything after it. That is far more than a sensible file needs, but automatically generated files on large sites do occasionally exceed it, at which point rules you believe are active are simply never read. If your file is anywhere near that size, the real fix is pattern consolidation rather than a longer file.

Free · 18 crawlers · runs in your browser

Robots.txt tester that shows which rule won

Google retired its own robots.txt tester, and most replacements give you a green tick with no explanation. A verdict you cannot audit is not much use when a single line is hiding half your catalogue.

This one implements the real matching rules — longest match wins, Allow beats Disallow on a tie, wildcards and end anchors — then shows you the group it selected, every rule that matched, and exactly why one of them decided the outcome. Including for the AI crawlers most testers ignore entirely.

Open the tester What robots.txt cannot do

1 Your robots.txt {{ sizeLabel }}

Load a preset

Path {{ resultPath }} {{ resultBot }} · {{ winLineLabel }} {{ resultLabel }}

The highlighted line is the one that decided this verdict for the URL selected in the matrix. Amber dots are rules that also matched but lost on specificity. Click anywhere in the file to edit it.

This example file has three real mistakes in it, deliberately, so the linter below has something to show you. Replace it with yours.

The version currently live

This is the check worth doing before you ship a robots.txt change, and the one nothing else offers. Cells that change verdict are ringed amber in the matrix.

What this change does {{ changeSummary }}

No URL you are testing changes verdict for any selected crawler. Either the edit is safe, or you are not testing the URLs it affects — add more paths above before trusting this.

2 URLs to test {{ urlCount }}

Homepage, sitemap, a stylesheet, a script, search, cart, login, a product, a post. Between them they catch most accidental blocks.

3 Crawlers {{ botCount }}

Results {{ tallyLabel }}

URL

Allowed Blocked Tap any cell for the rule trace

Rule trace {{ traceVerdict }}

{{ traceUrl }}

as {{ traceBot }}

Ambiguous input — {{ traceNote }}

Step 1 · group selection

Step 2 · rules that matched

L{{ m.line }} {{ m.text }} {{ m.len }}

Wins

No rule in that group matches this path. Nothing forbids it, so it is allowed by default.

Step 3 · why that one won

Linter {{ lintCount }}

Line {{ w.line }}: {{ w.snippet }}

Robots.txt does not deindex

Disallow stops crawling, not indexing. A blocked URL can still appear in search results if other pages link to it, listed without a snippet. To keep a page out of the index you need a noindex meta tag or header, which means the page has to be crawlable for Google to see it.

And it is not access control

It is a public file of polite requests. Well-behaved crawlers honour it; anything malicious reads it as a directory of what you would rather people did not see. Never list admin paths, staging URLs, or private directories in it. Use authentication for anything that actually matters.

The CDN caveat

A clean robots.txt does not prove a crawler can reach you. Cloudflare and similar services block AI bots above this layer, on by default in some plans. Check your edge rules and your server logs too — that side of it is covered on our answer engine optimisation page.

Under the hood

How a crawler decides whether a URL is blocked

Most robots.txt bugs are not typos. They come from assuming the file is read top to bottom like a script, when a crawler picks exactly one group and applies the most specific matching rule inside it.

Rule What it means Example that surprises people

One group only A crawler finds the most specific group matching its name and reads only that group. A Googlebot group means every rule under User-agent: * is ignored for Googlebot

Longest match wins Not first match. The rule with the longest matching path decides, regardless of order. Disallow: /blog then Allow: /blog/seo means /blog/seo is allowed

Ties go to Allow When Allow and Disallow match with identical length, the URL is allowed. Disallow: /a and Allow: /a resolves to allowed

Case sensitive Paths are compared exactly, including case. Disallow: /Search does not block /search

Wildcards are partial Asterisk and dollar are supported by Google and Bing, not by every crawler. Disallow: /*.pdf$ works for Google, ignored by many others

Errors change everything A 404 means allow all. A 5xx means Google treats the whole site as disallowed for up to 30 days. An outage on robots.txt can quietly deindex a site

500 KiB limit Google reads the first 500 kibibytes and ignores the rest. Auto-generated files on large platforms silently drop later rules

Worked cases

Four results people get wrong

Run each of these through the tester above and check your instinct against the answer.

Allowed, despite the Disallow

The Allow rule is longer, so it wins. Order in the file makes no difference.

robots.txt
  User-agent: *
  Disallow: /blog
  Allow: /blog/seo-guide

Testing /blog/seo-guide
  => ALLOWED (longest match)

Blocked, despite the Allow

The wildcard group is ignored entirely because Googlebot has its own group.

robots.txt
  User-agent: *
  Allow: /

  User-agent: Googlebot
  Disallow: /private

Testing /private for Googlebot
  => BLOCKED (own group wins)

Not blocked, because of case

Robots.txt matching is case sensitive even when the server is not.

robots.txt
  User-agent: *
  Disallow: /Search

Testing /search?q=shoes
  => ALLOWED (case mismatch)

Blocked, and still indexed

Blocking prevents fetching, not listing. This is the single most misunderstood behaviour in the file.

robots.txt
  User-agent: *
  Disallow: /old-page

Result
  Not crawled, but can still appear
  in results as a bare URL if linked

From real audits

What we find when we check a live file

These are the failures that cost real traffic, roughly in the order of how often we see them.

Staging rules shipped to production Disallow: / goes live with the launch and the site drops out of search within a fortnight. Make robots.txt environment-aware rather than a file someone remembers to edit.

Blocking CSS and JavaScript Google renders before judging. Block /assets/ and it sees an unstyled skeleton, which affects both mobile usability and what it believes the page contains.

Disallow used instead of noindex The crawler never fetches the page, so it never sees the noindex tag, and the URL lingers in the index with no description.

A blocked URL that holds backlinks A disallowed URL cannot pass equity onward, so every link pointing at it is wasted.

Rules that only exist on www Robots.txt applies to one host. The bare domain, www, and every subdomain each need their own file.

AdsBot ignored the site-wide block AdsBot-Google does not follow the wildcard group, which is usually a relief, but an explicit AdsBot rule can disapprove your landing pages.

Our stack

Billion Game runs your work through the same instruments the best in-house teams use.

Licences are on us, and every export we pull from them is yours to keep.

✎

Be wary of any agency whose reporting is a tool dashboard with their logo on it. Tools measure; they do not decide what matters, and every one of these will happily generate a hundred findings that change nothing. What you are paying for is the judgement about which three of those hundred are worth your developers’ time.

FAQ

Robots.txt questions

The matching rules are genuinely counter-intuitive, so these are worth reading once properly.

If pages are not indexed and robots.txt is clean, the cause is elsewhere: rendering, canonicals, crawl budget, or quality thresholds. That is a technical SEO audit rather than a robots.txt problem.

A correct robots.txt is one of about forty things that have to be right.

We check this alongside rendering, canonicals, crawl budget, and AI crawler access at the edge, then hand your developers tickets rather than a list of concerns.

Technical SEO audit More free tools

{{ col.title }}

{{ feature.title }}