CCBot: user agent, IP ranges and robots.txt

CCBot builds Common Crawl, a non-profit open repository of web crawl data. The archive is free to download and is one of the most common sources of AI training data.

Operator
Common Crawl
Type
Training · Collects content to train future models
robots.txt token
CCBot
Follows robots.txt
Yes
IP list
https://index.commoncrawl.org/ccbot.json
Official docs
commoncrawl.org

CCBot user agent

User agent

CCBot/2.0 (https://commoncrawl.org/faq/)

How to block or allow CCBot in robots.txt

Common Crawl documents "User-agent: CCBot / Disallow: /" as the way to opt out.

Block CCBot

User-agent: CCBot
Disallow: /

Allow CCBot

User-agent: CCBot
Allow: /

Put the rules in /robots.txt at the root of each host. Check the result with our AI Crawler Checker, which also catches firewall and CDN blocks that robots.txt doesn't show.

Should you block CCBot?

Blocking CCBot keeps future crawls out of the archive, which many AI companies train on, but copies already in older crawls stay there. Some AI search tools also use Common Crawl to discover pages.

How to verify real CCBot traffic

Reverse DNS on an IPv4 address should end in crawl.commoncrawl.org, and a forward lookup should match. Common Crawl warns that other crawlers falsely claim to be CCBot.

Loading the current list…

To check every request in your server log at once, use the free AI Bot Log Analyzer. It runs in your browser and flags requests that use a bot's name from outside its official IP list.

Frequently asked questions

What is CCBot?

CCBot builds Common Crawl, a non-profit open repository of web crawl data. The archive is free to download and is one of the most common sources of AI training data.

How do I block CCBot?

Add "User-agent: CCBot" followed by "Disallow: /" to your robots.txt. To block only part of the site, disallow those paths instead of "/". Common Crawl documents "User-agent: CCBot / Disallow: /" as the way to opt out.

Is a request claiming to be CCBot really from Common Crawl?

Not necessarily: anyone can copy a user agent. Reverse DNS on an IPv4 address should end in crawl.commoncrawl.org, and a forward lookup should match. Common Crawl warns that other crawlers falsely claim to be CCBot.

Checked against Common Crawl's documentation on 2026-11-06. Vendors change user agents and policies, so confirm in the official docs before relying on them.

Related crawlers

Is CCBot allowed on your site?

The free AI Crawler Checker tests robots.txt and firewall rules for 14 AI crawlers in one run.

Check my site