Free AI Crawler Checker
See which AI bots can read your site — and which ones your robots.txt turns away.
No signup. We don't store or log the domains you check.
Checking domain...
That's one signal. Your site has more.
The Website Health Check covers the big picture: Google Lighthouse speed scores plus DNS, SSL, email authentication, and AI access in one report.
Run the Website Health CheckScout can run your whole website with Surmado Sites, or list every fix first in a Site Audit.
How it works
Enter a domain and we fetch its robots.txt, then read it the way each AI crawler does: a bot with its own user-agent rule follows that rule, and a bot without one falls under the wildcard. You get one verdict per bot — OAI-SearchBot, Claude-SearchBot, PerplexityBot, GPTBot, ClaudeBot, Google-Extended, and Bytespider — with search crawlers, the ones that cite sources in live answers, separated from training crawlers.
When to use this
Run it after any robots.txt change, after a site migration or redesign, or whenever AI assistants seem to have gone quiet about your business. Blocking a search crawler removes you from AI answers without any error message on your side, so it's worth a check whenever visibility matters.
Frequently asked questions
Which AI crawlers does this check?
Seven bots in two groups. Search crawlers that fetch pages to answer live questions and cite sources: OAI-SearchBot (ChatGPT search), Claude-SearchBot, and PerplexityBot. Training crawlers that collect pages for future models: GPTBot, ClaudeBot, Google-Extended, and Bytespider.
Should I allow AI crawlers on my site?
For most businesses, allowing search crawlers is the clear win: they're how ChatGPT, Claude, and Perplexity cite you as a source. Training crawlers are a genuine choice — allowing them helps future models know your business; blocking them is a legitimate stance on content use. This tool shows what your current settings actually say. Our guide Which AI bots should my website allow? goes bot by bot, with a sample robots.txt.
Does blocking a crawler in robots.txt actually stop it?
The major crawlers this tool checks honor robots.txt. User-fetch agents — like ChatGPT-User, which fetches a page when a user pastes a link — don't reliably honor it, which is why we don't report a verdict for them.
How do I unblock ChatGPT search?
Open yourdomain.com/robots.txt and look for a Disallow: / that reaches OAI-SearchBot, the crawler behind ChatGPT search, under User-agent: OAI-SearchBot or User-agent: *. Delete it, or give the bot a group of its own, User-agent: OAI-SearchBot followed by Allow: /, which overrides the * rule. GPTBot only collects pages for training, so it doesn't decide whether you show up in ChatGPT search. OpenAI says a change can take about a day to register. Then run this check again, and read the rest of the file with the robots.txt checker. A firewall or CDN bot setting can also block ChatGPT, and robots.txt won't show it.
Check everything at once
One report covering speed, DNS, SSL, email authentication, and AI access — the signals that decide whether visitors, search engines, and AI assistants trust your site.
Run the Website Health Check