To get cited in AI search, start by allowing the search crawler: for ChatGPT search, OpenAI recommends allowing OAI-SearchBot in robots.txt and requests from its published IP ranges (Overview of OpenAI Crawlers). For Google, you can check whether your site is included in Search generative AI features in Search Console (Optimizing your website for generative AI features on Google Search). You can also limit how much content Google may use as direct input for AI Overviews and AI Mode through max-snippet (Robots meta tag, data-nosnippet, and X-Robots-Tag specifications).
Start with the groundwork in our earlier article: Is SEO dead now with AI?
How do I remove access and eligibility barriers to getting my website cited in Google AI Overviews and ChatGPT search?
Allowing a search crawler does not guarantee a citation: Google’s optimization guide says even meeting all requirements does not guarantee crawling, indexing or serving.
Keep URLs crawlable when you need Google to follow their indexing or serving directives. Google’s Robots meta tag, data-nosnippet, and X-Robots-Tag specifications explains that robots meta tags and X-Robots-Tag HTTP headers are discovered when a URL is crawled. If robots.txt blocks that crawl, Google will not find those rules and will ignore them.
Use the URL Inspection tool to examine the HTML Googlebot received when crawling the page, as directed in Google’s AI features and your website (AI features and your website). Check that your intended preview control is correct and visible in that received HTML, rather than relying only on what you intended to publish.
Keep ChatGPT search access separate from OpenAI model-training permission: you can allow OAI-SearchBot while disallowing GPTBot to indicate that crawled content should not be used for training its generative AI foundation models. OpenAI’s crawler documentation, accessed October 10, 2026, says search systems can take approximately 24 hours to adjust after a robots.txt update.
Do not use ChatGPT-User as the control for ChatGPT search eligibility (Overview of OpenAI Crawlers). OpenAI describes it as an agent for certain user-initiated actions, not automatic web crawling, and says it does not determine whether content may appear in Search. Use OAI-SearchBot in robots.txt to manage search opt-outs and automatic crawling instead. If you allow both OAI-SearchBot and GPTBot, OpenAI may use the results of a single crawl for search and training to avoid duplicative crawling. OpenAI publishes its search crawler’s IP addresses in searchbot.json, which provides the address ranges relevant to requests from OAI-SearchBot. Because ChatGPT-User actions are initiated by a user, OpenAI says robots.txt rules may not apply to those visits.
Can I limit Google's use of my content in AI answers without removing the page from ordinary search?
Google’s snippet restrictions affect ordinary search previews as well as AI features; they are not an AI-only opt-out.
Do not choose noindex if your goal is to keep the page in ordinary search results (Robots meta tag, data-nosnippet, and X-Robots-Tag specifications). Google’s robots specifications define noindex as an instruction not to show the page, media or resource in search results.
Choose nosnippet when you want to prevent the content from being used as direct input for AI Overviews and AI Mode (Robots meta tag, data-nosnippet, and X-Robots-Tag specifications). That instruction also prevents text snippets and video previews, although a static image thumbnail may remain when it provides a better user experience.
Choose max-snippet when your intended restriction is a maximum character length for the textual preview rather than preventing it altogether (Robots meta tag, data-nosnippet, and X-Robots-Tag specifications). The same setting limits how much content may be used as direct AI input, but it does not limit image or video previews. Its limit does not apply where you have separately granted permission, such as supplying in-page structured data or entering a licence agreement with Google.
Treat supplied structured data as a separate content-use decision, not something automatically covered by the surrounding text restriction (Robots meta tag, data-nosnippet, and X-Robots-Tag specifications). Google says robots meta tag limitations generally do not affect structured data, with exceptions for article.description and description values for other creative works. Use max-snippet to specify the maximum preview length based on those description values.
Modify structured-data types and values themselves to supply only the information you want available (Robots meta tag, data-nosnippet, and X-Robots-Tag specifications). Putting structured data inside a data-nosnippet element does not make it unavailable for search results.
Apply these decisions specifically to Google rather than assuming another search engine will interpret the rules identically. Supply a parseable number with max-snippet, because Google ignores the rule when that number is missing or cannot be parsed. The character limit applies to the textual snippet for a search result, and Google notes that a URL may appear as multiple results on a results page. Recipe structured data can remain eligible for a recipe carousel even when the page’s text preview would otherwise be limited. For that rich-result information, restricting the text preview with a robots meta tag is not a substitute for changing the structured data you supply.
For ChatGPT search, opting out of OAI-SearchBot prevents your site from appearing in search answers, although it can still appear as navigational links (Overview of OpenAI Crawlers).
Platform-specific access and control checks
| Check | Rule | What to verify |
|---|---|---|
| Google preview-control implementation | Preview controls must be correct and visible to Googlebot. | Use URL Inspection to examine the HTML Googlebot received. |
| Google crawl access for indexing and serving directives | Google discovers robots meta tags and X-Robots-Tag headers when a URL is crawled. | Check that URLs whose directives must be followed are not disallowed in robots.txt. |
| ChatGPT search access | Sites opted out of OAI-SearchBot do not appear in ChatGPT search answers, but can still appear as navigational links. | Check that robots.txt allows OAI-SearchBot and that requests from OpenAI's published searchbot IP ranges are allowed. |
| OpenAI search versus training settings | OAI-SearchBot and GPTBot settings are independent; search systems can take approximately 24 hours to adjust to a robots.txt update. | Check the two settings separately: allowing search does not require allowing GPTBot for training. |
| Google structured-data exposure | Structured data remains usable for search results when declared inside a data-nosnippet element. | Review the structured-data types and values themselves, adding or removing information to control what is available. |
Which Google report should I use to check generative AI visibility, and what should I distrust in third-party tools?
Use the Generative AI performance report in Search Console to measure your content’s performance in generative AI features on Google Search and Discover.
Treat third-party tools as workflow aids, not privileged access to Google’s ranking or AI systems. Be wary of tools that promise ranking success or claim to use internal Google metrics.
You can use a third-party tool when it helps your workflow, while evaluating its advice against Google’s official guidance.
Sources
Overview of OpenAI Crawlers, OpenAI, accessed 10 October 2026developers.openai.com
Optimizing your website for generative AI features on Google Search, Google Search Central, 10 July 2026developers.google.com
Robots meta tag, data-nosnippet, and X-Robots-Tag specifications, Google Search Central, 24 March 2026developers.google.com
AI features and your website, Google Search Central, 10 December 2025developers.google.com
