Googlebot: user agent, IP ranges and robots.txt
Googlebot is Google Search's main crawler. It also supplies the pages behind AI Overviews and AI Mode, since those are part of Google Search.
- Operator
- Type
- Search engine · Classic search crawler that also feeds AI Overviews or Copilot
- robots.txt token
Googlebot- Follows robots.txt
- Yes
- Official docs
- developers.google.com
Googlebot user agent
Googlebot Smartphone
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Googlebot Desktop
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36
W.X.Y.Z is the current Chrome version, which changes often. Google suggests wildcards when matching logs.
How to block or allow Googlebot in robots.txt
Google says its common crawlers always obey robots.txt when crawling automatically.
Block Googlebot
User-agent: Googlebot Disallow: /
Allow Googlebot
User-agent: Googlebot Allow: /
Put the rules in /robots.txt at the root of each host. Check the result with our AI Crawler Checker, which also catches firewall and CDN blocks that robots.txt doesn't show.
Should you block Googlebot?
Never block Googlebot on pages you want in Google. To keep a page out of search, allow crawling and use noindex.
How to verify real Googlebot traffic
Run a reverse DNS lookup on the IP: it must end in googlebot.com, google.com or googleusercontent.com. Then run a forward lookup on that hostname and confirm it returns the same IP. Or match the IP against Google's published ranges.
Loading the current list…
To check every request in your server log at once, use the free AI Bot Log Analyzer. It runs in your browser and flags requests that use a bot's name from outside its official IP list.
Frequently asked questions
What is Googlebot?
Googlebot is Google Search's main crawler. It also supplies the pages behind AI Overviews and AI Mode, since those are part of Google Search.
How do I block Googlebot?
Add "User-agent: Googlebot" followed by "Disallow: /" to your robots.txt. To block only part of the site, disallow those paths instead of "/". Google says its common crawlers always obey robots.txt when crawling automatically.
Is a request claiming to be Googlebot really from Google?
Not necessarily: anyone can copy a user agent. Run a reverse DNS lookup on the IP: it must end in googlebot.com, google.com or googleusercontent.com. Then run a forward lookup on that hostname and confirm it returns the same IP. Or match the IP against Google's published ranges.
Checked against Google's documentation on 2026-11-06. Vendors change user agents and policies, so confirm in the official docs before relying on them.
Related crawlers
- Google-ExtendedGoogle · Training
- BingbotMicrosoft · Search engine
- ApplebotApple · Search engine
- OAI-SearchBotOpenAI · AI search
- All AI crawlersThe full list with user agents and purposes.
The free AI Crawler Checker tests robots.txt and firewall rules for 14 AI crawlers in one run.