robots.txt tester
Paste the site's robots.txt (open https://example.com/robots.txt and copy it), pick a user agent and list the URLs or paths to test, one per line. Each line shows ALLOWED or BLOCKED and the rule that decided it. Matching follows RFC 9309 as Google applies it, and was checked against 13,548 cases from real robots.txt files with zero differences from Scrapy's Google-compatible parser. Nothing is uploaded.
Frequently asked questions
Which rule wins when Allow and Disallow both match?
The most specific one: the rule with the longest path. If they are equally long, Allow wins. The order of the lines does not matter.
Which group applies to my bot?
The group whose User-agent names the bot's product token (case-insensitive), e.g. Googlebot. All groups for that name are merged. Only if there is none does User-agent: * apply. Googlebot-Image, -Video and -News fall back to the Googlebot group, as Google documents.
Does Disallow stop a page from being indexed?
No. It stops crawling. A blocked URL can still be indexed from links, without its content. To keep a page out of search results, allow crawling and use a noindex meta tag or X-Robots-Tag header.
How do I block AI crawlers like GPTBot or ClaudeBot?
Add a group per crawler, e.g. 'User-agent: GPTBot' followed by 'Disallow: /'. Test it here with that user agent. Only crawlers that respect robots.txt are affected.
Why can't the tool fetch robots.txt for me?
Browsers block pages from reading files on other sites (CORS), and this page sends nothing to a server. Open the robots.txt in a new tab and paste it.