Bytespider: user agent, IP ranges and robots.txt

Bytespider is a crawler operated by ByteDance, the company behind TikTok and the Doubao AI assistant. It's commonly reported as collecting data for AI models.

Operator
ByteDance
Type
Training · Collects content to train future models
robots.txt token
Bytespider
Follows robots.txt
No published policy
IP list
not published

Bytespider user agent

ByteDance doesn't publish official documentation for Bytespider. Match on "Bytespider" in the user agent.

How to block or allow Bytespider in robots.txt

You can disallow it in robots.txt, but with no official documentation there's no published commitment on how it follows the rules. A firewall rule on the user agent is the reliable option.

Block Bytespider

User-agent: Bytespider
Disallow: /

Allow Bytespider

User-agent: Bytespider
Allow: /

Put the rules in /robots.txt at the root of each host. Check the result with our AI Crawler Checker, which also catches firewall and CDN blocks that robots.txt doesn't show.

Should you block Bytespider?

Block it in robots.txt and, if it keeps crawling, in your firewall or CDN. It doesn't power any search engine you'd lose traffic from in most markets.

How to verify real Bytespider traffic

ByteDance publishes no IP list or reverse DNS hostname, so requests can't be verified.

To check every request in your server log at once, use the free AI Bot Log Analyzer. It runs in your browser and flags requests that use a bot's name from outside its official IP list.

Frequently asked questions

What is Bytespider?

Bytespider is a crawler operated by ByteDance, the company behind TikTok and the Doubao AI assistant. It's commonly reported as collecting data for AI models.

How do I block Bytespider?

Add "User-agent: Bytespider" followed by "Disallow: /" to robots.txt, but it may not be respected: You can disallow it in robots.txt, but with no official documentation there's no published commitment on how it follows the rules. A firewall rule on the user agent is the reliable option. For a hard block, add a firewall or CDN rule that refuses requests whose user agent contains "Bytespider".

Is a request claiming to be Bytespider really from ByteDance?

You can't be sure. ByteDance publishes no IP list or reverse DNS hostname, so requests can't be verified. Treat the user agent as a claim, not proof.

Checked against ByteDance's documentation on 2026-11-06. Vendors change user agents and policies, so confirm in the official docs before relying on them.

Related crawlers

Is Bytespider allowed on your site?

The free AI Crawler Checker tests robots.txt and firewall rules for 14 AI crawlers in one run.

Check my site