Blocked by robots.txt: What It Means and How to Fix It [2026]

You open the Page indexing report in Search Console and find a long list of URLs marked as blocked by robots.txt. Some of them look harmless, like cart pages and internal search results. Others are product pages, blog posts or whole service sections that you very much want in Google.

That mix is normal. Blocking Googlebot with robots.txt is often deliberate and sensible, yet a single stray line can also hide the pages that earn your revenue. The trick is telling the two apart quickly, then fixing only the rules that are hurting you.

This guide explains what the status means, how to decide whether it matters, how to find the exact rule responsible, and how to fix it on a custom site, WordPress, Wix and Shopify. You will also learn why robots.txt is the wrong tool for removing a page from Google.

Key Takeaways

• The status means Googlebot was told not to crawl the URL. A rule in your robots.txt file matched the page, so Google did not fetch it.
• Often it is intended. Google's own help says URLs blocked by robots.txt are probably blocked on purpose, so only important pages need action.
• Find the rule with Google's tools. The URL Inspection tool confirms the block, and the robots.txt report shows the exact file Google fetched and when.
• robots.txt does not remove pages from Google. To keep a page out of search results, allow crawling and use noindex instead.
• Shopify uses robots.txt.liquid. You edit it through the theme code editor, and Shopify calls it an unsupported customisation that can cost you traffic if done badly.
• Validation takes time. Google says fix validation typically takes up to about two weeks.

What Does "Blocked by robots.txt" Mean?

Quick answer: "Blocked by robots.txt" means a rule in your site's robots.txt file stopped Googlebot from crawling that URL, so Google could not read the page. It is only a problem when the blocked URL is a page you want in search results. Find the matching rule with the URL Inspection tool and the robots.txt report, then remove or narrow it, and request a recrawl.

Search Console shows this status in two places. In the URL Inspection tool, the Page indexing section says Blocked by robots.txt. In Google's Page indexing report documentation, the same reason is listed as URL blocked by robots.txt, under the "Why pages aren't indexed" table.

The description is short. According to Google's Page indexing report help, the page "was blocked by your site's robots.txt file". Google also adds a warning that trips up many site owners: the block does not guarantee the page stays out of the index.

Crawling versus indexing

It helps to separate two jobs:

1. Crawling is Googlebot fetching the page and reading its content.
2. Indexing is Google storing the page so it can appear in search results.

robots.txt only controls the first job. Google's introduction to robots.txt says the file is used "mainly to avoid overloading your site with requests", and states plainly that it is not a mechanism for keeping a web page out of Google. If other sites link to a blocked URL, Google can still show the address, sometimes with anchor text from those links, but without a proper description.

Is "Blocked by robots.txt" Actually a Problem?

Frequently, no. Google's validation guidance notes that URLs blocked by robots.txt are "probably intentionally blocked" and asks you to use your judgement before fixing anything.

So start by sorting the example URLs into two piles: pages you meant to block, and pages you did not. The table below covers the patterns we see most often.

A quick sense check: if the blocked URLs appear in your XML sitemap, something is contradicting itself. Your sitemap asks Google to crawl the page while your robots.txt forbids it. Either the sitemap or the rule is wrong.

How to Find the Rule That Blocks a Page

Work through these steps in order. Each one narrows the problem further.

1. Review the example URLs in the Page indexing report

Open Indexing > Pages, then click the robots.txt reason in the "Why pages aren't indexed" table. The details page lists example URLs, capped at 1,000 rows per issue according to Google's help. Export the list and look for patterns: a shared folder, a parameter, or a subdomain.

If you submit sitemaps, use the All submitted pages filter at the top of the report. It shows only URLs listed in your sitemaps, which are the ones you have told Google matter.

2. Inspect one affected URL

Paste an affected URL into the URL Inspection tool at the top of Search Console. If the Page indexing section says Blocked by robots.txt, you have confirmed the problem. Google's "Unblock a page blocked by robots.txt" help page describes exactly this check.

Next, click Test live URL. The live test reflects your current robots.txt, so it tells you whether a fix you have already made is working.

3. Open the robots.txt report

In Search Console, go to Settings, then open the robots.txt report in the Crawling section. According to Google's robots.txt report help, the report shows the robots.txt files Google found for the top 20 hosts on your property, when each was last checked, its fetch status, its size and any parsing issues.

Two details make this report especially useful:

• Versions. Click a file, then Versions, to see every changed copy Google fetched in the last 30 days. If pages became blocked on a certain date, this shows you what changed.
• Hosts. Each host has its own file. shop.example.com/robots.txt and example.com/robots.txt are separate, so a blocked subdomain may be using a file you never check.

4. Test the rule against the URL

Paste your robots.txt into our free robots.txt tester with the blocked URL and user agent, and it will show which line matches. Developers can also run Google's open source robots.txt library locally, which Google's help says is the same library used in Google Search.

How Google decides which rule wins

Many accidental blocks come from misreading how rules combine. According to Google's robots.txt specification:

• The most specific rule wins, measured by the length of the rule's path.
• When an allow and a disallow rule are equally specific, Google uses the least restrictive one.
• Paths are case-sensitive, so Disallow: /Blog does not block /blog.
• Google supports two wildcards: * for any run of characters, and $ for the end of the URL.
• Google ignores content beyond a 500 KiB file size limit.
• Google generally caches robots.txt for up to 24 hours, so a fix is not seen instantly.

Here is how those rules play out on real patterns:

How to Fix Blocked by robots.txt, by Cause

Once you know the matching rule, the cause is usually one of these, and each has a direct fix.

After any edit, open the file in your browser at yourdomain.com/robots.txt to confirm the live version changed. Caching layers sometimes serve an old copy for hours.

One related trap is worth knowing. If your robots.txt file itself returns a server error, Google's specification says it stops crawling the site for the first 12 hours while it keeps retrying, then uses the last good copy for up to 30 days. That is not the same status, yet it can look like a site-wide block, so check the fetch status in the robots.txt report too.

Want a Page Out of Google? Use noindex, Not robots.txt

This is the most common misunderstanding behind the status. Blocking a page does not remove it from Google's index.

Google's documentation on blocking indexing with noindex is explicit: for noindex to work, the page must not be blocked by robots.txt, because a crawler that cannot fetch the page never sees the noindex rule. Google's Page indexing help gives the same instruction: to make sure a page is not indexed, remove the robots.txt block and use noindex.

So the right tool depends on your goal:

• To save crawling on low-value URLs, such as endless filter combinations, use robots.txt.
• To keep a page out of search results, allow crawling and add a noindex meta tag or X-Robots-Tag header.
• To keep something private, put it behind a password.

There is a practical reason this matters. Google's crawl budget guidance notes that blocked URLs stay in the crawl queue much longer and are recrawled when the block is removed. A block hides the page from Googlebot, but it does not make Google forget the URL. If your blocked pages are not being crawled for other reasons too, our guide on fixing Discovered, currently not indexed covers the crawl side in more depth.

Platform Notes: Shopify, WordPress and Wix

Hosted platforms control robots.txt differently, and Google's own help admits it cannot give exact steps for every host. Here is where each file lives.

Blocked by robots.txt on Shopify

Every Shopify store has a default robots.txt. According to Shopify's help on editing robots.txt.liquid, the default file blocks admin, cart, checkout, account and order pages, plus filtered collection URLs (/collections/*+*) and sorted ones (/collections/*sort_by*). Those blocks are deliberate, so a Shopify store will always show some URLs as blocked by robots.txt.

To customise it:

1. From your Shopify admin, go to Online Store.
2. For your theme, open the menu and click Edit code.
3. Click Add a new template, then select robots.
4. Click Create template, make your changes with Liquid, and save.

Shopify's help carries a strong warning. It calls this an unsupported customisation that Shopify Support cannot help with, and says incorrect use "can result in loss of all traffic". Shopify recommends adding or removing directives with Liquid rather than replacing the template with plain text, so the file keeps receiving Shopify's future updates.

One detail many stores miss: Shopify notes that an older robots.txt.liquid template can keep rules that the current default no longer has, such as Disallow: /search and Disallow: /policies/. If your policy pages show as blocked, an outdated template may be the reason. Deleting the template restores the current default. For the wider picture on store indexing, see our guide on getting Google to index your Shopify website.

WordPress

WordPress serves a virtual robots.txt when no physical file exists at the site root. To change it, upload a physical robots.txt or use an SEO plugin's editor, such as Yoast SEO > Tools > File editor. A physical file overrides the virtual one, so check the server when plugin edits do not seem to apply.

Wix

Wix sites have a built-in editor. Go to SEO & GEO in your dashboard, scroll to Tools and settings, and open the Robots.txt Editor. If an edit went wrong, the same screen has a Reset to Default button.

How to Validate the Fix

Once the rule is fixed, tell Google so it rechecks sooner.

1. Confirm the live file. Load yourdomain.com/robots.txt and check your change is there.
2. Request a robots.txt recrawl. In the robots.txt report, open the menu next to the file and click Request a recrawl. Google's help suggests this when you have unblocked important URLs, while noting it does not guarantee an immediate recrawl of those URLs.
3. Run a live test. In URL Inspection, click Test live URL on an affected page. It should no longer report a robots.txt block.
4. Request indexing for your most important pages.
5. Click Validate fix on the issue in the Page indexing report, if the affected URLs are ones you want indexed.

Be patient with the last step. Google's Page indexing help says validation "typically takes up to about two weeks", and in some cases much longer. If your pages are crawled but still not indexed afterwards, that is a different problem, usually about quality or duplication rather than access.

Our technical SEO services include a full crawl-control review, covering robots.txt, noindex, canonicals and sitemaps together, because these signals so often contradict one another.

How Billion Game Approaches It

When we audit this status, we start with the sitemap filter, because a blocked URL that is also in the sitemap is a mixed signal. Then we check the robots.txt report's version history for the date the block appeared, which usually points to the release or plugin that caused it.

If you would rather hand the whole report to a specialist team, our Google Search Console issue fixing service diagnoses each status, applies the fixes and runs validation, with reporting from your own Search Console data.

Frequently Asked Questions

Final Thoughts

Most sites should have some URLs blocked by robots.txt. Carts, checkouts, internal search and filter combinations do not belong in Google, and blocking them keeps crawling focused on pages that matter.

The problem starts when a rule catches revenue pages, or when robots.txt is used to hide something it cannot hide. Confirm the block with URL Inspection, trace it through the robots.txt report, narrow the rule, and use noindex for anything that must stay out of results. Handled that way, blocked by robots.txt becomes a status you control rather than one that controls your traffic.

About the author

Ganesh Shanmugam

Ganesh Shanmugam is an SEO and AEO consultant based in Chennai, India, and the founder of Billion Game — a boutique agency serving clients across the US, UK, Australia, Canada, and Ireland. He previously held SEO roles at 7 Eagles and Digital Scholar before going independent, and brings 4+ years of hands-on experience

More from Billion Game →