Instantly fetch and parse the robots.txt file for any domain. See exactly which crawl directives apply to each user-agent — Googlebot, Bingbot, or any custom crawler — and test whether a specific URL path is allowed or blocked.

A misconfigured robots.txt is one of the most common technical SEO mistakes. A single wrong directive can accidentally block your entire site from search engines. This tool helps you catch those issues instantly.

What you can do with this tool

Fetch the robots.txt file for any domain and view its raw content
See parsed crawl rules per user-agent (Googlebot, Bingbot, *)
Test whether a specific URL path is allowed or disallowed for crawling
Identify sitemap declarations referenced in robots.txt
Detect common mistakes: blocking /, missing sitemap reference, conflicting rules

How to test a robots.txt file

Enter the domain URL (e.g. https://example.com)
Click Fetch to retrieve the robots.txt file
Browse the parsed rules by user-agent
Enter a specific path (e.g. /admin) to test if it's allowed or blocked

FAQ

Does robots.txt block indexing?

Robots.txt controls crawling, not indexing. A URL can still appear in search results even if it's disallowed in robots.txt, if Google discovered it through links. To block indexing, use a noindex meta tag or HTTP header instead.

Are robots.txt rules mandatory for crawlers?

No. Robots.txt is a courtesy standard (the Robots Exclusion Protocol). Legitimate crawlers like Googlebot respect it, but malicious bots may ignore it entirely.