Robots.txt Tester
Check whether search engines and AI crawlers can access any URL on your site — test your robots.txt against real Google matching rules in seconds. Free.
- ✓ Real Google-spec matching
- ✓ Test any crawler (incl. AI bots)
- ✓ Fetch or paste robots.txt
- ✓ Instant allowed/blocked result
One wrong line in your robots.txt can accidentally block Google from your most important pages — and you might not notice for months. This free tester lets you fetch your live robots.txt (or paste one), then check whether any URL is allowed or blocked for a specific crawler, using the same wildcard and most-specific-rule logic Google actually applies.
Catch costly mistakes
Find out instantly if you’re blocking pages you meant to keep crawlable — before it hurts your rankings.
Test any crawler
Check Googlebot, Bingbot and AI crawlers like GPTBot, ClaudeBot, PerplexityBot and Google-Extended.
Accurate matching
Supports wildcards (*), end-anchors ($) and Google’s longest-match-wins rule.
Test Your Robots.txt
Fetch your robots.txt (or paste it), then test any URL and crawler.
Matching follows Google’s rules (wildcards *, end-anchor $, most-specific rule wins). Free tool.
How It Works
Add your robots.txt
Enter your website and click “Fetch robots.txt” — or paste your rules directly.
Choose a URL & crawler
Enter the URL or path to test and pick the crawler (e.g. Googlebot or GPTBot).
See the result
Instantly see whether it’s allowed or blocked, and exactly which rule decided it.
What Is a Robots.txt File?
robots.txt is a plain-text file at the root of your site (e.g. example.com/robots.txt) that tells crawlers which parts of your site they may or may not access. It uses User-agent lines to target specific crawlers and Allow / Disallow rules to control access. Getting it wrong can hide your site from search engines entirely.
How Robots.txt Matching Works
For a given crawler, Google picks the most specific matching User-agent group, then applies the most specific (longest) matching rule — and if an Allow and Disallow rule are equally specific, Allow wins. Rules support the * wildcard (any characters) and $ (end of URL). This tester replicates that logic so your results match reality.
Robots.txt & AI Crawlers
As AI search grows, you may want to control access for AI crawlers too — GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot and Google-Extended. Test how your rules apply to each, so you deliberately choose what AI engines can and can’t use. If you want to appear in AI answers, don’t block them — see our GEO service.
Explore More Free Tools & Services
Frequently Asked Questions
Is this robots.txt tester free?
Yes — testing is free and unlimited, and the matching runs in your browser. Fetching a live robots.txt has a generous daily fair-use limit.
What does “blocked” mean?
It means the crawler you selected is not allowed to crawl that URL under your current robots.txt rules. Blocked pages generally won’t be crawled (though they can still be indexed if linked elsewhere).
Can I test AI crawlers like GPTBot?
Yes — the crawler dropdown includes GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more, so you can see exactly how your rules apply to AI bots.
Does robots.txt stop a page being indexed?
Not by itself. Robots.txt controls crawling, not indexing — a blocked page can still appear in results if other sites link to it. To keep a page out of the index, use a noindex meta tag instead.
Where should my robots.txt live?
At the root of your domain: https://yourdomain.com/robots.txt. It only applies to that exact host and protocol.
Why Do It Manually?
A clean robots.txt is just one piece of technical SEO. Let our team audit and fix crawling, indexing and everything else — so search engines see your best pages.
Browse All Free Tools →Check My Ranking