Glossary
What is ClaudeBot?
ClaudeBot is Anthropic's web crawler that collects public pages that may be used to train Claude models. Anthropic also runs Claude-SearchBot, which indexes pages to improve Claude's search answers, and Claude-User, which fetches pages when a Claude user asks. All respect robots.txt.
Anthropic's bots
| Bot | Purpose |
|---|---|
| ClaudeBot | Collect web content that may be used for model training |
| Claude-SearchBot | Improve the relevance of search results in Claude |
| Claude-User | Fetch pages when a user asks Claude to access them |
Anthropic's help center documents the user agents and says the bots honour robots.txt and don't bypass CAPTCHAs or log-ins.
robots.txt example
User-agent: ClaudeBot
Disallow: /private/
User-agent: Claude-SearchBot
Allow: /
Anthropic also supports the non-standard Crawl-delay directive to slow crawling.
Which to allow
As with OpenAI's bots: allow the search and user bots if you want Claude to find and cite you; decide on ClaudeBot based on your training-data policy.
Example
A developer-tools company allows all three bots and publishes an llms.txt. When developers ask Claude how to set up its SDK, Claude can fetch and use the current docs.
Checking access
Look for ClaudeBot, Claude-SearchBot and Claude-User in your server or CDN logs. If they never appear, check robots.txt groups and any CDN setting that blocks AI crawlers. A site can allow a bot in robots.txt and still block it at the network edge.
FAQ
Does ClaudeBot respect robots.txt?
Anthropic says yes.
What's the difference between ClaudeBot and Claude-User?
ClaudeBot crawls for training; Claude-User fetches a page because a user asked.
Can I slow ClaudeBot down?
Anthropic supports Crawl-delay in robots.txt.
Does blocking ClaudeBot stop Claude from reading my page when a user shares it?
That request comes from Claude-User, which has its own entry; set each one as you prefer.