AI crawler list: user agents of every AI bot
Every major AI crawler, grouped by what it does, with its user agent, robots.txt token and whether its owner publishes an IP list. Open a bot for the robots.txt rules to block or allow it and how to verify its traffic.
AI search crawlers
These index pages so AI assistants can find and cite them. Blocking them costs you visibility in AI answers. (AI search: indexes pages so an assistant can cite them in answers.)
| Crawler | Company | User agent | IP list |
|---|---|---|---|
OAI-SearchBot | OpenAI | Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot | Yes |
Claude-SearchBot | Anthropic | not published | Yes |
PerplexityBot | Perplexity | Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot) | Yes |
DuckAssistBot | DuckDuckGo | DuckAssistBot/1.2; (+http://duckduckgo.com/duckassistbot.html) | Yes |
User-triggered fetchers
These open a page because a person asked an assistant about it. Several don't follow robots.txt, since a human made the request. (User request: fetches a page live because someone asked an assistant about it.)
| Crawler | Company | User agent | IP list |
|---|---|---|---|
ChatGPT-User | OpenAI | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot | Yes |
Claude-User | Anthropic | not published | Yes |
Perplexity-User | Perplexity | Mozilla/5.0 (compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user) | Yes |
meta-externalfetcher | Meta | meta-externalfetcher/1.1 | No |
MistralAI-User | Mistral | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-User/1.0; +https://docs.mistral.ai/robots) | Yes |
AI training crawlers and tokens
These collect content to train models, or are robots.txt tokens that control training. Blocking them is a legitimate choice and doesn't affect AI search. (Training: collects content to train future models.)
| Crawler | Company | User agent | IP list |
|---|---|---|---|
GPTBot | OpenAI | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot | Yes |
ClaudeBot | Anthropic | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com) | Yes |
Google-Extended | robots.txt token only | No | |
Applebot-Extended | Apple | robots.txt token only | No |
CCBot | Common Crawl | CCBot/2.0 (https://commoncrawl.org/faq/) | Yes |
meta-externalagent | Meta | meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler) | No |
Bytespider | ByteDance | not published | No |
Mixed-purpose crawlers
Used for several products, including AI. (Other AI: research, product development or agent crawling.)
| Crawler | Company | User agent | IP list |
|---|---|---|---|
Amazonbot | Amazon | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1) Chrome/W.X.Y.Z Safari/537.36 | Yes |
Search engine crawlers
Classic search crawlers whose indexes also feed AI features like Google's AI Overviews and Microsoft Copilot. (Search engine: classic search crawler that also feeds ai overviews or copilot.)
| Crawler | Company | User agent | IP list |
|---|---|---|---|
Googlebot | Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | Yes | |
Bingbot | Microsoft | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/W.X.Y.Z Safari/537.36 | Yes |
Applebot | Apple | Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot) | Yes |
Block AI training, keep AI search
Most sites want to be cited by ChatGPT, Claude and Perplexity but may not want to feed model training. This robots.txt does that for the main vendors:
User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot User-agent: meta-externalagent User-agent: Bytespider Disallow: / User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot Allow: /
robots.txt only works for bots that follow it, and firewall settings such as Cloudflare's AI bot blocking can refuse the search bots anyway. The free AI Crawler Checker tests both, and the AI Bot Log Analyzer shows which bots really visit.
User agents and policies checked against each vendor's documentation on 2026-11-06.
Check robots.txt and firewall rules for 14 AI crawlers in one free run.