Question 1

Does robots.txt stop a page being indexed?

Accepted Answer

No. Robots.txt controls crawling, not indexing. A disallowed URL can still appear in results if other sites link to it, because Google can index a URL it has never fetched. To keep a page out of the index, allow the crawl and use a noindex meta robots tag, or protect it behind authentication.

Question 2

What happens if I do not have a robots.txt file at all?

Accepted Answer

Nothing bad. A missing robots.txt returns 404 and crawlers treat the entire site as allowed, which is the correct default for most small sites. The file only becomes necessary when you have URL patterns worth keeping crawlers away from, or a sitemap worth declaring.

Question 3

Can robots.txt slow down or speed up indexing?

Accepted Answer

Indirectly, yes. Blocking low-value URL patterns returns crawl budget to pages that can rank, which on large sites measurably speeds up how quickly new pages are discovered. On a site under a few thousand URLs the effect is usually negligible.

Question 4

Do I need a separate robots.txt for each subdomain?

Accepted Answer

Yes. Robots.txt applies to exactly one host, so shop.example.com, blog.example.com and example.com each need their own file at their own root. The same applies across protocols, though in practice everything should redirect to HTTPS anyway.

Question 5

How do I test my robots.txt before deploying it?

Accepted Answer

Paste it into our robots.txt tester with a specific URL and it will tell you whether that URL is blocked and by which rule. Google Search Console also has a robots.txt report showing the file as Google last fetched it, which is worth checking after any change.

Question 6

Should I block AI crawlers like GPTBot?

Accepted Answer

It depends what you want. Blocking GPTBot and ClaudeBot removes your content from model training, but blocking OAI-SearchBot and PerplexityBot also removes you from the answers those assistants give live, which is a visibility decision rather than a copyright one. Most businesses that want AI recommendations should allow the search-time crawlers and decide separately about training crawlers.

Question 7

Where does robots.txt have to live?

Accepted Answer

At the root of the host, exactly at /robots.txt, served over the same protocol and subdomain as the pages it governs. A file at /folder/robots.txt is ignored, and a robots.txt on www does not apply to a bare domain or to a separate subdomain such as shop or blog.

Question 8

What is the difference between Disallow and noindex?

Accepted Answer

Disallow in robots.txt tells a crawler not to fetch the URL. Noindex in a meta tag or HTTP header tells it not to keep the page in the index. They conflict when used together: a disallowed page can never be fetched, so its noindex tag is never seen, and the page can linger in the index indefinitely.

Question 9

Does Google support crawl-delay?

Accepted Answer

No. Google ignores the crawl-delay directive entirely and manages fetch rate through its own algorithms and Search Console settings. Bing and several other crawlers do respect it, so it is still worth including if a specific bot is overloading your server.

Question 10

Should I block scrapers like AhrefsBot and SemrushBot?

Accepted Answer

Only if server load is a genuine problem. Blocking them hides your backlink profile from those tools, which also means your competitors cannot study it, but it means you lose visibility into your own data too. A crawl delay is usually the better compromise.

Question 11

Does robots.txt affect Google Ads or Merchant Center?

Accepted Answer

Yes, and this catches people out. AdsBot-Google does not follow the wildcard User-agent group, so a site-wide disallow will not block it, but an explicit AdsBot rule will and can disapprove your landing pages. Merchant Center similarly needs to fetch product URLs to validate them.

Question 12

Can I use regular expressions in robots.txt?

Accepted Answer

No, but Google and Bing support two wildcard characters: an asterisk matching any sequence, and a dollar sign anchoring the end of a URL. So a rule ending in .pdf and a dollar sign blocks PDFs. Most other crawlers support neither, so treat wildcards as a Google and Bing feature only.

Question 13

How long does a robots.txt change take to have an effect?

Accepted Answer

Google caches robots.txt for around twenty-four hours, so most changes take effect within a day. Removing a disallow does not immediately index the pages, though: they still have to be crawled and assessed, which typically takes days to weeks depending on the site.

Question 14

Should the file be at robots.txt or /robots.txt?

Accepted Answer

It must be served at the root path exactly, so https://example.com/robots.txt. Crawlers do not look anywhere else. A file inside a folder, or one served under a different filename, is simply never requested.

Question 15

Does blocking a URL remove it from Google?

Accepted Answer

Not reliably, and not quickly. If the page is already indexed, blocking the crawl freezes it there, because Google cannot fetch it to discover it has changed. To remove a page, allow the crawl and serve a noindex tag or a 410 status, then use the Search Console removal tool for anything urgent.

{{ col.title }}

{{ feature.title }}

Robots.txt generator

Every robots.txt directive, and who actually supports it

Six robots.txt files you can copy

Ecommerce store

WordPress site

Staging or development

Allow AI search, block AI training

News or publisher

Aggressive bot, everything else open

How a crawler actually reads this file

The mistakes we find most often

Robots.txt is a crawl instruction, not a lock

Billion Game runs your work through the same instruments the best in-house teams use.