Claude-SearchBot: user agent, IP ranges and robots.txt

Claude-SearchBot crawls the web to improve the quality of search results in Claude. Disallowing it stops Anthropic indexing your content for search.

Operator
Anthropic
Type
AI search · Indexes pages so an assistant can cite them in answers
robots.txt token
Claude-SearchBot
Follows robots.txt
Yes
IP list
https://claude.com/crawling/bots.json
Official docs
support.claude.com

Claude-SearchBot user agent

Anthropic publishes the robots.txt token but not the full user agent string. Match on the token, which appears in the user agent of every request.

How to block or allow Claude-SearchBot in robots.txt

Anthropic says its bots respect robots.txt; disallowing Claude-SearchBot keeps your content out of its search index.

Block Claude-SearchBot

User-agent: Claude-SearchBot
Disallow: /

Allow Claude-SearchBot

User-agent: Claude-SearchBot
Allow: /

Put the rules in /robots.txt at the root of each host. Check the result with our AI Crawler Checker, which also catches firewall and CDN blocks that robots.txt doesn't show.

Should you block Claude-SearchBot?

Allow it if you want Claude to find and cite your pages. Blocking it can lower your visibility in Claude's answers, even if you allow Claude-User.

How to verify real Claude-SearchBot traffic

Check the request IP against Anthropic's published list.

Loading the current list…

To check every request in your server log at once, use the free AI Bot Log Analyzer. It runs in your browser and flags requests that use a bot's name from outside its official IP list.

Frequently asked questions

What is Claude-SearchBot?

Claude-SearchBot crawls the web to improve the quality of search results in Claude. Disallowing it stops Anthropic indexing your content for search.

How do I block Claude-SearchBot?

Add "User-agent: Claude-SearchBot" followed by "Disallow: /" to your robots.txt. To block only part of the site, disallow those paths instead of "/". Anthropic says its bots respect robots.txt; disallowing Claude-SearchBot keeps your content out of its search index.

Is a request claiming to be Claude-SearchBot really from Anthropic?

Not necessarily: anyone can copy a user agent. Check the request IP against Anthropic's published list.

Checked against Anthropic's documentation on 2026-11-06. Vendors change user agents and policies, so confirm in the official docs before relying on them.

Related crawlers

Is Claude-SearchBot allowed on your site?

The free AI Crawler Checker tests robots.txt and firewall rules for 14 AI crawlers in one run.

Check my site