Google-Extended: user agent, IP ranges and robots.txt
Google-Extended is a robots.txt control token, not a separate crawler. It lets you decide whether content Google crawls may be used to train Gemini models and to ground answers in Gemini Apps and Vertex AI.
- Operator
- Type
- Training · Collects content to train future models
- robots.txt token
Google-Extended- Follows robots.txt
- Control token only (no crawling)
- IP list
- not published
- Official docs
- developers.google.com
Google-Extended user agent
Google-Extended has no user agent of its own: Google crawls with its normal user agents, and the token only works as a robots.txt control. You won't see it in your logs.
How to block or allow Google-Extended in robots.txt
Only used in robots.txt. Google says it does not affect a site's inclusion in Google Search and isn't a ranking signal.
Block Google-Extended
User-agent: Google-Extended Disallow: /
Allow Google-Extended
User-agent: Google-Extended Allow: /
Put the rules in /robots.txt at the root of each host. Check the result with our AI Crawler Checker, which also catches firewall and CDN blocks that robots.txt doesn't show.
Should you block Google-Extended?
Disallow Google-Extended to keep your content out of Gemini training and grounding without any effect on Google Search. AI Overviews are part of Google Search, so this token doesn't remove you from them.
How to verify real Google-Extended traffic
Nothing to verify: requests come from Googlebot and other Google crawlers.
Frequently asked questions
What is Google-Extended?
Google-Extended is a robots.txt control token, not a separate crawler. It lets you decide whether content Google crawls may be used to train Gemini models and to ground answers in Gemini Apps and Vertex AI.
How do I use Google-Extended in robots.txt?
Add "User-agent: Google-Extended" followed by "Disallow: /" to opt out. It doesn't crawl, so there's nothing to see in your logs. Only used in robots.txt. Google says it does not affect a site's inclusion in Google Search and isn't a ranking signal.
Does blocking Google-Extended remove me from AI Overviews?
No. AI Overviews are part of Google Search, which Google-Extended doesn't affect. Pages are kept out of AI Overviews the same way as out of search snippets, with noindex or nosnippet.
Checked against Google's documentation on 2026-11-06. Vendors change user agents and policies, so confirm in the official docs before relying on them.
Related crawlers
- GooglebotGoogle · Search engine
- Applebot-ExtendedApple · Training
- GPTBotOpenAI · Training
- ClaudeBotAnthropic · Training
- All AI crawlersThe full list with user agents and purposes.
The free AI Crawler Checker tests robots.txt and firewall rules for 14 AI crawlers in one run.