Which AI Bots Should My Website Allow? A Plain-English Guide

Part of AI for Small Business, our plain-English guide to what AI can and can't do for your business.

Last verified: August 2026

Every AI company runs more than one bot, and they don't all do the same job. This page groups them by what actually happens to your site, not by logo, so you can decide what to allow instead of guessing.

This guide covers AI-specific bots only, not regular Google indexing. If you're not sure your site shows up on Google at all, start with our calm diagnostic for why your site isn't showing up on Google, then come back here.

Three job types, not one big "AI bot" category

Every AI bot that visits your site falls into one of three jobs. Knowing which job a bot has tells you what allowing or blocking it actually does.

Search bots

These govern whether an AI platform's search or answer feature may crawl and index your site for its answers. OAI-SearchBot feeds ChatGPT's search feature. Claude-SearchBot feeds Claude, and Anthropic warns that blocking it may reduce your visibility in Claude's answers. PerplexityBot feeds Perplexity.

User-fetch agents

These visit your site the moment a specific person asks an AI assistant about you, in real time. ChatGPT-User, Claude-User, and Perplexity-User each do this job for their own platform.

Training bots

These crawl your site to help shape what a future version of the model knows. GPTBot does this for OpenAI, ClaudeBot for Anthropic, and Google-Extended for Google's Gemini. None of the three change anything today.

What won't fix it: allowing every bot doesn't guarantee you'll be cited. It only removes a technical blocker. The content still has to be good enough, and specific enough, for a model to want to cite it. Our guide to getting found by ChatGPT covers that half of the problem.

The bots, by vendor

Here's what each company actually runs, and the one detail per vendor worth knowing before you touch robots.txt.

Which bot handles each job, by vendor
Vendor	Search bot	User-fetch bot	Training bot
OpenAI	OAI-SearchBot	ChatGPT-User	GPTBot
Anthropic	Claude-SearchBot	Claude-User	ClaudeBot
Perplexity	PerplexityBot	Perplexity-User	None published
Google	Regular Googlebot, not a separate AI bot	None named	Google-Extended

OpenAI: notes that when you allow both OAI-SearchBot and GPTBot, a single crawl may serve both purposes.
Anthropic: all three of its bots, Claude-SearchBot, Claude-User, and ClaudeBot, honor robots.txt.
Perplexity: documents that Perplexity-User, the fetch triggered by a real person's question, generally does not honor robots.txt, even though PerplexityBot does.
Google: Google-Extended is a robots.txt token, not a separate crawler. It controls Gemini's training and grounding only. Google states plainly that it does not impact a site's inclusion or ranking in Google Search. AI Overviews draw on the normal Googlebot index, so Google-Extended isn't a lever for AI Overviews either.

What won't fix it: assuming one vendor's rules apply to another. Each company sets its own bot behavior, and the differences above are real, not fine print.

Two things most owners get wrong

First: robots.txt is default-allow. If you've never touched the file, you're very likely blocking nothing on purpose. Most sites that do block AI bots didn't choose to. A security plugin, a firewall, or CDN bot protection turned it on by default, often without telling the owner.

Second: user-fetch agents don't all behave the same way. Claude-User respects robots.txt. ChatGPT-User and Perplexity-User may not. Don't treat a robots.txt disallow line as a reliable way to stop a user-fetch agent from reaching a page. It's a request, not a lock.

What won't fix it: assuming your current setup is doing what you think it is. The only way to know what's actually blocked is to check it directly.

How do you actually check?

Trust Check reads your live robots.txt and tells you which of these bots your policy allows or blocks. It covers the search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and the training bots (GPTBot, ClaudeBot, Google-Extended).

Run Trust Check

What it can't see: Trust Check reads your published policy, not vendor behavior. A firewall or CDN can still block a bot your robots.txt allows, and a clean policy doesn't make a platform start citing you. Our AI Visibility check is what tests whether platforms actually do.

A sample robots.txt: visible everywhere, training your choice

This file allows every search and user-fetch bot, which removes the robots-policy blocker on being found and cited. It leaves the three training bots as an explicit choice instead of a leftover default. Copy it, adjust the training section to match your own answer, then save it at yourdomain.com/robots.txt.

# Search bots: govern indexing for AI answers
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# User-fetch agents: visit when someone asks about you directly
User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Perplexity-User
Allow: /

# Training bots: shape what future models know. Allow or disallow is your call.
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Allow: /

What won't fix it: copying this file doesn't guarantee you'll be crawled or cited soon. It removes one possible blocker. It can also take time for a platform to notice and recrawl a changed file.

Fix it yourself

The steps above are the honest version, in order. Jump back to the steps and work through them. No account, no email, nothing to buy.

Never deal with this again

Hand Surmado the site. We rebuild it, run it, and keep everything you've built along the way.

$99/mo. Hosting, maintenance, and updates included. We rebuild your site free. You see it before you pay anything.

El Tianguis went from invisible to a 67% AI visibility score. Read the case study

Common questions

Does blocking GPTBot remove me from ChatGPT?

No. GPTBot is OpenAI's training bot. Blocking it only keeps your content out of future model training. The bot ChatGPT search uses to index sites is a separate one, OAI-SearchBot, with its own robots.txt control.

Does Google-Extended affect my Google ranking?

No. Google-Extended controls only Gemini's training and grounding. Google states directly that it does not impact a site's inclusion or ranking in Google Search, and AI Overviews draw from the normal Googlebot index, not from Google-Extended.

Should a small business block AI training bots?

It depends on what you're optimizing for, and there's no single right answer. Blocking training bots keeps new content out of future model training, which some owners want on principle. It won't remove you from AI search results today, since that's a different bot's job, and it won't undo any training that already happened on older content. Treat it as a deliberate tradeoff, not a default setting.

What is llms.txt?

A machine-readable summary file some AI tools read, similar in spirit to a sitemap but written for language models instead of search crawlers. Google has said its search does not use it. Surmado ships one on its own site. It's a reasonable thing to add, but it doesn't replace the per-bot robots.txt settings above.