Free Robots.txt Checker
See what a site's robots.txt allows in — and what it locks out.
No signup. We don't store or log the domains you check.
Checking domain...
That's one signal. Your site has more.
The Website Health Check covers the big picture: Google Lighthouse speed scores plus DNS, SSL, email authentication, and AI access in one report.
Run the Website Health CheckHow it works
We fetch the site's robots.txt and read the rules the way crawlers do: which user-agents are named, what each is disallowed from, and whether a wildcard rule quietly blocks everything. Verdicts cover the file itself, search engines like Googlebot, and AI crawler access.
When to use this
After a site launch — staging blocks left in robots.txt are one of the most common ways a new site vanishes from search. Also before and after SEO work, and whenever pages drop out of Google for no visible reason.
Frequently asked questions
What is robots.txt?
A plain-text file at yourdomain.com/robots.txt that tells crawlers which parts of the site they may visit. It's a convention, not a lock — but legitimate crawlers honor it, including the ones this tool checks.
Can robots.txt hurt my Google ranking?
A wrong rule can remove you from Google entirely — the classic case is a Disallow: / left over from staging. That's exactly what this tool flags: wildcard blocks, blocked search engines, and blocked AI crawlers.
What's the difference between this and the AI Crawler Checker?
This page reads the whole file: search engines, wildcard rules, and an overall AI verdict. The AI Crawler Checker answers one question in more depth — which specific AI bots can read your site, one verdict per bot.
Check everything at once
One report covering speed, DNS, SSL, email authentication, and AI access — the signals that decide whether visitors, search engines, and AI assistants trust your site.
Run the Website Health Check