Claude-User: user agent, IP ranges and robots.txt

When people ask Claude questions, it may open websites with the Claude-User agent to answer them. Disallowing it stops Claude retrieving your content for those questions.

Operator
Anthropic
Type
User request · Fetches a page live because someone asked an assistant about it
robots.txt token
Claude-User
Follows robots.txt
Yes
IP list
https://claude.com/crawling/bots.json
Official docs
support.claude.com

Claude-User user agent

Anthropic publishes the robots.txt token but not the full user agent string. Match on the token, which appears in the user agent of every request.

How to block or allow Claude-User in robots.txt

Anthropic documents a robots.txt opt-out for Claude-User: disallowing it prevents Claude retrieving your content in response to a user query.

Block Claude-User

User-agent: Claude-User
Disallow: /

Allow Claude-User

User-agent: Claude-User
Allow: /

Put the rules in /robots.txt at the root of each host. Check the result with our AI Crawler Checker, which also catches firewall and CDN blocks that robots.txt doesn't show.

Should you block Claude-User?

Usually allow it. Each visit is a person asking Claude about your page, and blocking it means Claude answers without your content.

How to verify real Claude-User traffic

Check the request IP against Anthropic's published list.

Loading the current list…

To check every request in your server log at once, use the free AI Bot Log Analyzer. It runs in your browser and flags requests that use a bot's name from outside its official IP list.

Frequently asked questions

What is Claude-User?

When people ask Claude questions, it may open websites with the Claude-User agent to answer them. Disallowing it stops Claude retrieving your content for those questions.

How do I block Claude-User?

Add "User-agent: Claude-User" followed by "Disallow: /" to your robots.txt. To block only part of the site, disallow those paths instead of "/". Anthropic documents a robots.txt opt-out for Claude-User: disallowing it prevents Claude retrieving your content in response to a user query.

Is a request claiming to be Claude-User really from Anthropic?

Not necessarily: anyone can copy a user agent. Check the request IP against Anthropic's published list.

Checked against Anthropic's documentation on 2026-11-06. Vendors change user agents and policies, so confirm in the official docs before relying on them.

Related crawlers

Is Claude-User allowed on your site?

The free AI Crawler Checker tests robots.txt and firewall rules for 14 AI crawlers in one run.

Check my site