Quick Answer: Google-Extended is a robots.txt token that controls whether Google may use your crawled content to train and ground Gemini models. It has no user agent of its own; Googlebot does the fetching and the token only sets permission. Google states it does not affect inclusion or ranking in Google Search, whose AI features follow the Googlebot rules.
Google-Extended is one of the most misread lines in a robots.txt file. Site owners add it to keep their pages out of AI Overviews, or delete it to get into them, and neither does anything, because the token governs a different set of products entirely.
Google describes it in one paragraph of its common crawlers documentation, last updated 14 July 2026. This page unpacks that paragraph: what the token covers, why it is not a crawler, what it does not touch, and how to write the rule.
What Google-Extended controls and what it leaves alone
Google-Extended controls two uses of your content: training future Gemini models and grounding Gemini answers, in Gemini Apps and on Vertex AI. It leaves Google Search alone, including indexing, ranking and every AI feature inside Search.
The two uses, in Google's words
Google's crawler doc defines the token in a single sentence, and the training half comes first. It calls Google-Extended "a standalone product token that web publishers can use to manage whether content Google crawls from their sites may be used for training future generations of Gemini models that power Gemini Apps and Vertex AI API for Gemini".
The same sentence adds grounding. Google defines that in the doc as "providing content from the Google Search index to the model at prompt time to improve factuality and relevancy", in Gemini Apps and Grounding with Google Search on Vertex AI.
So the token has a training half and a grounding half, and one rule sets both. There is no separate token for "train but do not ground", which matters for the decision later on this page.
What it does not touch
The doc's closing sentence rules out any Search effect. It reads: "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search." Blocking it costs you nothing in the ten blue links and gains you nothing there either.
At a glance
| Question | Answer from Google's documentation |
|---|---|
| Is it a crawler? | No. It has no separate HTTP user agent string |
| Who fetches the pages? | Existing Google user agents, chiefly Googlebot |
| What it governs | Training of future Gemini models; grounding in Gemini Apps and Vertex AI |
| What it does not govern | Inclusion or ranking in Google Search |
| Where it applies | Only in robots.txt, as a User-agent token |
| First announced | 28 September 2023, then covering Bard and Vertex AI |
Why Google-Extended is a token and not a crawler
Google-Extended never sends a request, so you will never see it in a server log. The crawler doc states: "Google-Extended doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity."
What that means in practice
Googlebot fetches the page once, for Search. Whether the fetched copy may also feed Gemini training and grounding is then decided by whether your robots.txt carries a Google-Extended group that disallows the path. The token is a permission flag read against a crawl that would have happened anyway.
Three consequences follow. You cannot block Google-Extended by IP or by user-agent filtering at the firewall, because there is nothing to match, and you cannot measure how often "it crawls", because it does not. A rule for Googlebot does not imply a rule for Google-Extended, or the reverse; each token is read on its own.
How it differs from the crawlers around it
The same doc lists Google-CloudVertexBot, which does have a user agent string. It covers "crawls requested by the site owners' for building Vertex AI Agents" and has "no effect on Google Search or other products". That is the opposite design: a real fetcher, triggered by a customer, with no bearing on Gemini training, and the two are confused only because both mention Vertex AI.
Does Google-Extended affect AI Overviews or AI Mode
No. Google's AI features and your website doc says the Googlebot rule is the control for AI features in Search. It points to Google-Extended only for "some of Google's other systems".
The exact wording
The doc's own sentence ties the AI features to the Googlebot rule. It states: "AI is built into Search and integral to how Search functions, which is why robots.txt directives for Googlebot is the control for site owners to manage access to how their sites are crawled for Search."
It then names the snippet controls and hands off to the token: "To limit the information shown from your pages in Search, use nosnippet, data-nosnippet, max-snippet, or noindex controls. To limit AI training and grounding in some of Google's other systems, read more about Google-Extended."
The same doc says there are "no additional requirements to appear in AI Overviews or AI Mode" beyond being indexed and eligible for a snippet. A page that Googlebot may crawl and that carries no snippet restriction is eligible for both features whether or not Google-Extended is disallowed.
What this means for a brand
If your goal is to appear in AI Overviews or in AI Mode, Google-Extended is irrelevant to it. The levers are the ones that apply to Search generally: crawlable pages, an indexable snippet, and content worth citing.
If your goal is the opposite, keeping content out of AI Overviews, Google-Extended does not do that either. The controls Google names are nosnippet, data-nosnippet, max-snippet and noindex, each of which also limits or removes the page from ordinary Search results.
How to write the Google-Extended rule in robots.txt
The rule is a standard robots.txt group with Google-Extended as the user-agent token, and the same Allow and Disallow syntax as any other group. Google's own example allows one path and disallows its parent:
User-agent: Google-Extended
Allow: /archive/1Q84
Disallow: /archive/
The three common configurations
To opt the whole site out of Gemini training and grounding while leaving Search untouched:
User-agent: Google-Extended
Disallow: /
To opt out one section, such as paid or member content, and leave the rest in:
User-agent: Google-Extended
Disallow: /members/
Disallow: /reports/
To state explicit permission, which is the default anyway but makes the intent visible to anyone auditing the file:
User-agent: Google-Extended
Allow: /
Mistakes that make the rule do nothing
A group must have its own User-agent: Google-Extended line; adding the token to a comma-separated list on another group's line is not valid robots.txt. A Googlebot Disallow does not carry over, and a wildcard User-agent: * block is overridden for Google-Extended by any group that names it specifically, which is how robots.txt directives for AI bots behave in general.
Robots.txt is read per host, so a rule on www.example.com says nothing about docs.example.com. And because Google's common crawlers "always obey robots.txt rules when crawling automatically", the file is the whole mechanism; there is no meta tag or HTTP header equivalent for this token.
Should you block Google-Extended
Block Google-Extended only if you have decided that neither Gemini training nor Gemini grounding should use your content, because one rule switches off both. For a brand that wants to be recommended in Gemini answers, blocking it works against that goal.
The grounding cost
Grounding is the mechanism by which Gemini Apps and Vertex AI pull content from the Search index at prompt time to answer a question. Disallowing Google-Extended tells Google that your pages may not be used for that. A brand working on visibility in Gemini is asking to be grounded on, so the disallow rule removes exactly the surface it is trying to win.
The training question
The training half is a policy decision rather than a marketing one. Publishers that license their archives, or that have a contractual reason to keep content out of model training, have a clear case to disallow. A commercial brand usually benefits from a model that has read accurate descriptions of what it sells, because that is the model's fallback when no live page is fetched.
A rule of thumb
If you would be pleased to see Gemini describe and cite your pages, leave Google-Extended allowed. If you have a licensing reason to opt out, disallow it and accept that Gemini answers will be grounded on other people's descriptions of you. Either way, the decision belongs in an AI visibility audit alongside the OpenAI, Anthropic and Perplexity tokens, each of which splits training from answering differently.
Google-Extended vs Googlebot vs Google-CloudVertexBot
The three names cover three jobs: Googlebot fetches for Search, Google-Extended sets Gemini permission on those fetches, and Google-CloudVertexBot fetches for a customer's own Vertex AI agent. Each has its own robots.txt token and none inherits the others' rules.
| Token | Sends requests? | What a Disallow does | Affects Search? |
|---|---|---|---|
| Googlebot | Yes | Removes pages from crawling for Search, Discover and all Search features including AI Overviews and AI Mode | Yes |
| Google-Extended | No | Withholds crawled content from Gemini training and grounding | No |
| Google-CloudVertexBot | Yes | Stops site-owner-requested crawls for Vertex AI Agents | No |
How to verify the fetches you do see
Because Google-Extended never appears in logs, the only Google requests you can verify are the real crawlers. Google's doc says common crawlers "generally crawl from the IP ranges published in the common-crawlers.json object", with a reverse DNS mask matching crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com. Check those before treating any Googlebot request as genuine, since the user agent string can be spoofed.
Once the token is set deliberately, the rest of answer engine optimization proceeds as if it were not there. It is a permission line for one company's models, not a switch for whether AI engines can find you.