A webmaster can allow OAI-SearchBot in the robots.txt file in order to appear in ChatGPT search results, and that setting is independent of the others (1). OpenAI states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. OpenAI says a webmaster can allow OAI-SearchBot in robots.txt in order to appear in search results, and it offers the OAI-SearchBot and GPTBot robots.txt tags so webmasters can manage how their sites work with AI.

How do I make my site eligible to be named and cited in ChatGPT search answers?

Start by allowing OAI-SearchBot in your site's robots.txt file (1). OpenAI uses this crawler to surface websites in the search features of ChatGPT, and it recommends allowing its published IP ranges as well.

After you change robots.txt, allow about 24 hours for OpenAI's systems to adjust before you judge the result (1). Because OpenAI's search results can take about 24 hours to reflect a robots.txt update, you should not judge the change before that window has passed. OpenAI says each robots.txt setting is independent and that a webmaster can allow OAI-SearchBot to appear in search results while disallowing GPTBot, which describes what these settings manage.

Which OpenAI crawlers should I allow or block, and what does each setting control?

Each OpenAI crawler setting is independent of the others (1). You can allow OAI-SearchBot so you appear in search results while disallowing GPTBot, which signals that your crawled content should not be used for training OpenAI's generative AI foundation models.

GPTBot is used to crawl content that may be used in training OpenAI's generative AI foundation models, and disallowing it indicates that a site's content should not be used for that training (1).

ChatGPT-User is different: it visits a page when a person asks ChatGPT or a Custom GPT a question, and it is not used for automatic crawling (1). Because those visits are user-initiated, robots.txt rules may not apply to them. If your site allows both OAI-SearchBot and GPTBot, OpenAI may use the results of a single crawl for both purposes to avoid duplicative crawling. OpenAI describes ChatGPT-User as not used to determine whether content may appear in Search, so for search opt-outs you should manage OAI-SearchBot in robots.txt.

Web Crawlers Used by OpenAI and How to Control Them in robots.txt

CrawlerWhat it is used forWhat to do in robots.txt
OAI-SearchBotSurfacing websites in ChatGPT's search featuresAllow it in robots.txt and allow OpenAI's published IP ranges
GPTBotCrawling content that may be used to train OpenAI's generative AI foundation modelsOptional: disallow it to signal content should not be used for training; the setting is independent of OAI-SearchBot

How can I check that OpenAI's search crawler is reaching my site?

Compare the requests in your server logs with OpenAI's published details (1). OAI-SearchBot identifies itself in its user-agent string, and OpenAI publishes its IP addresses at openai.com/searchbot.json.

When OAI-SearchBot fetches robots.txt, it may add a robots.txt marker to its user-agent string (1). That marker lets you separate robots.txt requests from other requests, which matters when your logs do not include paths. The OAI-SearchBot user-agent string includes its version number, which OpenAI notes may change, so match on the bot name rather than an exact version.

Sources

  1. Overview of OpenAI Crawlers, OpenAI, accessed 8 November 2026developers.openai.com