Robots.txt Tester
Check whether robots.txt permits crawling of a URL, which is a narrower question than whether the page can appear in search results.
- ✓ Any URL
- ✓ Rule matching shown
- ✓ Free and unlimited
- ✓ Crawling, not indexing
This tests one thing: whether your robots.txt allows a crawler to fetch a given URL. A page can pass this test and still be absent from results because of a noindex tag, a canonical pointing elsewhere, a redirect, or simply not having been crawled yet. Knowing what the test does not cover is most of its value.
Rule matching made visible
Which directive applies to the URL you entered.
Location pages checkable
The pages most often blocked by a leftover rule.
Scope stated
Crawling only, with the other causes named.
Test Your Robots.txt
Fetch your robots.txt (or paste it), then test any URL and crawler.
Matching follows Google’s rules (wildcards *, end-anchor $, most-specific rule wins). Free tool.
How It Works
Add your robots.txt
Enter your website and click “Fetch robots.txt”, or paste your rules directly.
Choose a URL & crawler
Enter the URL or path to test and pick the crawler (e.g. Googlebot or GPTBot).
See the result
Instantly see whether it’s allowed or blocked, and exactly which rule decided it.
Blocked is not the same as excluded
A disallow rule stops a page being crawled. It does not stop it being indexed, a blocked URL can still appear in results with no description underneath, which is usually a worse outcome than either alternative. To keep a page out of results, use a noindex tag and allow the crawl, so the tag can actually be read.
The failure that lasts for years
The recurring local-business problem is a rule left over from development that quietly blocks a location page, or a whole directory of them. There is no symptom. Nothing breaks, no error appears, and the pages simply never rank. Testing the location pages specifically, not just the homepage, is how it gets found.
What else to check
If a page passes here and still does not appear, check four things. The meta robots tag, the canonical, any redirect in the chain, and whether the URL is in your sitemap. This test rules out one cause; it does not rule out the others, and treating a pass as a clean bill of health is the common misreading.
Explore More Free Tools & Services
Frequently Asked Questions
Does blocking a page remove it from Google?
No. A blocked page can still be indexed without a description. Noindex is the correct instrument for exclusion.
Why can't I use both disallow and noindex?
Because a blocked page is never fetched, so the noindex tag is never read. Allow the crawl if you want the tag to work.
What should I test?
Your location and service pages specifically. Leftover development rules affect those far more often than the homepage.
When should I re-check?
After any migration, rebuild or theme change. Those are when stray rules reappear.
My page passes but does not rank. Why?
This test covers crawling only. Check meta robots, canonicals, redirects and whether the page is in your sitemap.
Crawlable is not the same as visible
A page that can be crawled may still be nowhere in the results customers see. Measuring your actual position across your trading area is what distinguishes a technical problem from a visibility one.
Browse All Free Tools →Check My RankingRobots.txt Tester
The robots file tells crawlers where they may go. It is a few lines of text and it is capable of removing a whole site from search with one of them.
The classic failure is a staging site rule that goes live with the site. Everything looks fine, nothing gets indexed, and the cause sits in a file nobody has opened since launch.
It is worth being clear about what it does not do. Blocking a page stops it being crawled, not indexed. A blocked page can still appear in results, with no description, which is the worst of both.
How to use it
- Load your robots file and read it as it stands.
- Test the URLs that matter. Home, services, location pages.
- Look for a blanket disallow rule left over from development.
- Use noindex, not robots, for pages you want kept out of results.
Why it matters
This is a two minute check that occasionally explains months of confusion. It is worth doing after any site move or redesign, without waiting for a reason.
Questions people ask
Can robots.txt remove a page from Google?
Not reliably. It blocks crawling. A blocked page can still be listed without a description. Use noindex to keep something out.
What is the most common mistake?
A disallow all rule from a staging site shipped to production. It hides everything.
Do I need one at all?
Not necessarily. An absent file is safer than a wrong one.