ClaudeBot: user agent, IP ranges and robots.txt
ClaudeBot collects public web content that may be used to train Anthropic's Claude models. Anthropic says its bots honor robots.txt.
- Operator
- Anthropic
- Type
- Training · Collects content to train future models
- robots.txt token
ClaudeBot- Follows robots.txt
- Yes
- Official docs
- support.claude.com
ClaudeBot user agent
Commonly seen in server logs
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)
Anthropic publishes the robots.txt token but not the full user agent string. Match on the token, which appears in the user agent of every request.
How to block or allow ClaudeBot in robots.txt
Anthropic says its bots respect robots.txt, and ClaudeBot also supports the non-standard Crawl-delay directive.
Block ClaudeBot
User-agent: ClaudeBot Disallow: /
Allow ClaudeBot
User-agent: ClaudeBot Allow: /
Put the rules in /robots.txt at the root of each host. Check the result with our AI Crawler Checker, which also catches firewall and CDN blocks that robots.txt doesn't show.
Should you block ClaudeBot?
Block ClaudeBot if you don't want your content used for training. It doesn't affect Claude's search answers, which use Claude-SearchBot and Claude-User. If it crawls too fast, add a Crawl-delay instead of blocking it.
How to verify real ClaudeBot traffic
Check the request IP against Anthropic's published list. Anthropic notes that blocking by IP isn't a reliable opt-out; use robots.txt.
Loading the current list…
To check every request in your server log at once, use the free AI Bot Log Analyzer. It runs in your browser and flags requests that use a bot's name from outside its official IP list.
Frequently asked questions
What is ClaudeBot?
ClaudeBot collects public web content that may be used to train Anthropic's Claude models. Anthropic says its bots honor robots.txt.
How do I block ClaudeBot?
Add "User-agent: ClaudeBot" followed by "Disallow: /" to your robots.txt. To block only part of the site, disallow those paths instead of "/". Anthropic says its bots respect robots.txt, and ClaudeBot also supports the non-standard Crawl-delay directive.
Is a request claiming to be ClaudeBot really from Anthropic?
Not necessarily: anyone can copy a user agent. Check the request IP against Anthropic's published list. Anthropic notes that blocking by IP isn't a reliable opt-out; use robots.txt.
How do I slow ClaudeBot down?
Add a Crawl-delay for it in robots.txt, for example "User-agent: ClaudeBot" followed by "Crawl-delay: 1". Anthropic supports this, unlike Google and Apple.
Checked against Anthropic's documentation on 2026-11-06. Vendors change user agents and policies, so confirm in the official docs before relying on them.
Related crawlers
- Claude-SearchBotAnthropic · AI search
- Claude-UserAnthropic · User request
- GPTBotOpenAI · Training
- CCBotCommon Crawl · Training
- All AI crawlersThe full list with user agents and purposes.
The free AI Crawler Checker tests robots.txt and firewall rules for 14 AI crawlers in one run.