robots.txt Tester & Validator
Fetch any site’s robots.txt, read its rules grouped by user-agent, and test whether a given path is allowed or blocked for Googlebot or any crawler — using the same longest-match logic Google does.
How it works
Fetch robots.txt
Enter a domain and the tester loads the public robots.txt file server-side.
Test a path
Choose a crawler and path to check against the site rules.
Read the matching rule
See the allow or disallow line that decided the result.
Frequently asked questions
What happens if a site has no robots.txt?
If robots.txt is missing, crawlers may access public URLs by default. A file is still useful for sitemap discovery and crawl guidance.
Does Disallow remove a page from Google?
No. Disallow controls crawling, not indexing. Use noindex on the page if you want it removed from search results.
Which rule wins in robots.txt?
The most specific matching rule wins. If allow and disallow rules have the same length, the allow rule is treated as the winner.
Built by 1MarketingTool
A free utility from the team behind 1MarketingTool — a local-first desktop app for rank tracking, keyword research, AI content and AI search optimization. BYOK, zero cloud.