Glossary
What is an AI crawler?
An AI crawler is a bot that fetches web pages for an AI company, either to collect training data, to build a search index for AI answers, or to fetch a page live when a user asks. Examples include GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and CCBot. Each can be allowed or blocked in robots.txt.
Three kinds of AI bot
| Kind | What it does | Examples |
|---|---|---|
| Training | Collects pages to train future models | GPTBot, ClaudeBot, CCBot |
| AI search | Indexes pages to cite in AI search answers | OAI-SearchBot, PerplexityBot, Claude-SearchBot |
| User-triggered | Fetches a page when a user asks the assistant to | ChatGPT-User, Claude-User, Perplexity-User |
Google-Extended is a special case: it's not a separate crawler but a robots.txt token that controls whether content Googlebot already crawls can be used for Gemini training and grounding.
Allow or block?
The decision depends on what you want:
- Want to be cited in AI answers? Allow the search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) and user fetchers.
- Don't want content used for training? Block training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended) and keep search ones allowed.
- Want everything? Allow all; it's the default if you don't mention them.
For a product that wants to be recommended, blocking AI search crawlers is usually self-defeating.
Beyond robots.txt
Some CDNs, including Cloudflare, offer settings that block AI bots at the network level, sometimes on by default for new sites. Check those too: a site can allow a bot in robots.txt and still block it at the edge. User-triggered fetchers may not follow robots.txt in the same way as crawlers, since they act for a user; each company documents its policy.
How to check what's crawling you
Server or CDN logs show user agents. Verify that a bot is genuine using the IP ranges each company publishes, since user agents can be faked.
Example
A SaaS blocks all bots with "AI" in the name after reading about training. Months later it notices it never appears as a source in ChatGPT or Perplexity answers. It unblocks OAI-SearchBot and PerplexityBot, keeps GPTBot blocked, and starts appearing as a cited source within weeks.
FAQ
Does blocking GPTBot remove me from ChatGPT search?
No. ChatGPT search uses OAI-SearchBot. GPTBot is for training. Block them separately as you prefer.
Do AI crawlers follow robots.txt?
The major companies say their crawlers do. User-triggered fetchers can follow different rules; check each company's documentation.
How do I know if an AI crawler visited my site?
Check server or CDN logs for their user agents and verify IPs against published ranges.
Sources
- OpenAI: Overview of OpenAI crawlers
- Anthropic: Does Anthropic crawl data from the web
- Perplexity: Perplexity crawlers
- Google Search Central: Google's common crawlers