Crawler policy
Does Google-Extended block AI Overviews, or do you need the Search Console control?
Google now documents a Search generative AI control. It belongs beside the crawler and snippet rules, with a different job from each.

Google-Extended does not remove a site from AI Overviews. For that choice, Google now documents a Search generative AI control in Search Console. Its help page says the control rolled out worldwide on August 31, 2026. I would check that setting before reaching for a Googlebot block. Search generative AI control.
An owner can want ordinary Search traffic, decline appearance in generative Search answers, and make a further choice about model training. The controls let those intentions differ. The work is to put each decision in the right place and preserve a receipt of the effective setting.
Method & limits
Official Google documentation was checked October 21, 2026. Google’s help page states that the Search generative AI control rolled out worldwide on August 31, 2026. No authenticated Search Console property or setting was inspected or changed. The property hierarchy and publisher decisions below are worked examples.
Which control matches your goal?
Google’s current control inventory links the Search Console setting alongside robots and page directives. Its crawler reference describes Google-Extended as a product-use token without a separate request User-Agent. Google’s sharing controls and crawler reference support the map below.
Think about the consequence for a reader. A publisher declining generative Search presentation may still want someone searching for its title to find the ordinary result. A Googlebot disallow would change the crawl path supporting Search much more broadly.
| Goal | Relevant control | Scope or consequence |
|---|---|---|
| Manage appearance in specified generative Search features | Search Console: Search generative AI | AI Overviews, AI Mode and generative AI features in Discover |
| Manage specified Gemini training and grounding uses | Google-Extended in robots.txt | Product-use preference; does not control Search inclusion or ranking |
| Manage automatic crawling for Google Search | Googlebot in robots.txt | Search crawling, including its AI features |
| Remove a page from Google Search results | noindex, visible to the crawler | Broader than opting out of generative Search |
| Limit page previews or selected text | nosnippet, max-snippet or data-nosnippet | Page/text controls with effects beyond a Search Console opt-out |
Where is the Search Console setting?
Google places it under Settings → Search generative AI. Its documented choices are inclusion, exclusion, or inheritance from a parent property. Exclusion concerns the site’s links and content in the listed generative features; the setting does not control training or other Search ranking and inclusion signals. Setting and scope.
For a publication that wants ordinary search results but declines the specified AI features, I would open the intended property, inspect its effective choice, and record whether it is inherited. The owner can then make a targeted change with the affected domain or path visible before saving.
Check the identity at the top of the property selector. A setting saved for an old hostname, a narrower blog prefix or a property belonging to another site is a poor receipt for the intended domain. The useful record pairs the property identifier with the saved choice and check time.
Why can a child property disagree with the domain?
Imagine example.com permits inclusion while https://example.com/reporting/ has a manually configured exclusion. A further property at https://example.com/reporting/press/ can have its own explicit choice. This is a worked hierarchy: the nearest configured parent matters unless the child has overridden inheritance. Google’s inheritance examples.
The maintenance risk is easy to miss. An operator changes the domain setting and assumes every section followed, while an earlier child override survives. Before a broad change, inventory the relevant properties and inspect any explicit child choices. Afterward, read the effective setting back for the sections that matter.
Keep a small table with property, parent, effective choice and whether the choice is manual. It is more useful than a screenshot cropped so tightly that the property identifier disappears.
What does Google-Extended still do?
Google-Extended covers the future Gemini training and grounding uses specified in Google’s crawler reference. The reference says ordinary Google agents perform the requests, so there is no Google-Extended network visit to count. Google-Extended definition.
The example below expresses a site-wide preference for those documented uses. Merge it with the existing file and test the combined policy. It leaves the Search Console decision to the property setting and leaves Googlebot’s crawl policy to its own matching rules.
Additional robots.txt group. This is not a replacement for the site’s existing file or a Search Console setting.
User-agent: Google-Extended
Disallow: /When are page and text directives the better fit?
A publisher might want one passage excluded from snippets while the rest of a page remains available, or want an entire page removed from Search. Those are editorial choices at a finer or broader level than the generative Search setting. Google documents data-nosnippet for selected text, nosnippet for page previews and noindex for Search exclusion. Googlebot must be able to retrieve the directive. Page and text directive reference.
For example, an editor could wrap a sensitive public quotation in a supported data-nosnippet element after deciding that a public quotation is appropriate at all. The attribute governs Google’s presentation of that text. Confidential material belongs behind authorization; preview controls do not turn a public response into a private one.
This is why I would avoid adding noindex to every article merely to decline AI Overviews. The selected control should express the publisher’s actual distribution decision. The broader crawler-policy guide supplies the surrounding agent map.
How do you verify the change?
Keep the saved Search Console choice with its property and inheritance state. For robots or page-directive changes, retain the publicly delivered file or HTML and use URL Inspection where appropriate to check what Google received. The raw HTML and rendered DOM guide helps when the directive visible in a browser differs from the response you captured.
Google describes a propagation period for the Search Console change and warns that cached content can take longer. Record when the setting changed before inspecting subsequent results. One query or one screenshot is too narrow to judge a whole site’s outcome.
I would close the configuration work when the right property shows the intended effective choice and the supporting files still express the owner’s other decisions. Any claim about the effect on traffic needs a dated measurement window after that receipt.
Sources
- Google Search Console Help — Search generative AI control
Google
Verified Oct 21, 2026
Official page explicitly states worldwide rollout as of August 31, 2026. No publication or modification date is displayed; rollout date is an event date, not an invented publication date. Setting, scope, timing and inheritance checked.
- Google Search Central — Control what you share with Google Search
Google · Updated October 21, 2026
Verified Oct 21, 2026
- Google Crawling Infrastructure — List of common crawlers / Google-Extended
Google · Updated October 15, 2026
Verified Oct 21, 2026
- Google Search Central — Robots meta, data-nosnippet and X-Robots-Tag
Google · Updated March 24, 2026
Verified Oct 21, 2026
- Google Search Central — AI features and your website
Google · Updated December 10, 2025
Verified Oct 21, 2026
Older page checked for crawl/snippet context. It does not list the later Search Console control; newer control-specific help and current sharing inventory govern the control inventory in this article.