Free AI Crawler Checker
See which AI bots can read your site — and which ones your robots.txt turns away.
No signup. We don't store or log the domains you check.
Checking domain...
That's one signal. Your site has more.
The Website Health Check covers the big picture: Google Lighthouse speed scores plus DNS, SSL, email authentication, and AI access in one report.
Run the Website Health CheckHow it works
Enter a domain and we fetch its robots.txt, then read it the way each AI crawler does: a bot with its own user-agent rule follows that rule, and a bot without one falls under the wildcard. You get one verdict per bot — OAI-SearchBot, Claude-SearchBot, PerplexityBot, GPTBot, ClaudeBot, Google-Extended, and Bytespider — with search crawlers, the ones that cite sources in live answers, separated from training crawlers.
When to use this
Run it after any robots.txt change, after a site migration or redesign, or whenever AI assistants seem to have gone quiet about your business. Blocking a search crawler removes you from AI answers without any error message on your side, so it's worth a check whenever visibility matters.
Frequently asked questions
Which AI crawlers does this check?
Seven bots in two groups. Search crawlers that fetch pages to answer live questions and cite sources: OAI-SearchBot (ChatGPT search), Claude-SearchBot, and PerplexityBot. Training crawlers that collect pages for future models: GPTBot, ClaudeBot, Google-Extended, and Bytespider.
Should I allow AI crawlers on my site?
For most businesses, allowing search crawlers is the clear win: they're how ChatGPT, Claude, and Perplexity cite you as a source. Training crawlers are a genuine choice — allowing them helps future models know your business; blocking them is a legitimate stance on content use. This tool shows what your current settings actually say.
Does blocking a crawler in robots.txt actually stop it?
The major crawlers this tool checks honor robots.txt. User-fetch agents — like ChatGPT-User, which fetches a page when a user pastes a link — don't reliably honor it, which is why we don't report a verdict for them.
Check everything at once
One report covering speed, DNS, SSL, email authentication, and AI access — the signals that decide whether visitors, search engines, and AI assistants trust your site.
Run the Website Health Check