AI Bots & Crawlers

OpenAI crawlers explained: GPTBot, OAI-SearchBot, ChatGPT-User and OAI-AdsBot

Understand OpenAI’s search, training, user-requested and advertising bots, with practical robots.txt examples, verification guidance and troubleshooting.

Four separate paths connect a website to a magnifying glass, notebooks, a bookmark and an advertising sign.

Key takeaways

  • OpenAI documents four distinct identities: OAI-SearchBot for search, GPTBot for potential training collection, ChatGPT-User for certain user actions, and OAI-AdsBot for submitted advertising pages. [1]
  • You can permit search crawling while declining GPTBot collection. These preferences operate independently. [1]
  • A declared user agent is a clue, not proof of who sent a request. Check the relevant published address ranges; signed Cloud browser requests have a separate verification process. [1] [4]
  • Robots rules, successful website access, answer citations, and human visits answer different questions. Investigate each with its own evidence.

OpenAI uses different web agents for different jobs. OAI-SearchBot helps discover pages for ChatGPT search. GPTBot collects content that may contribute to foundation-model training. ChatGPT-User retrieves pages for certain user-requested actions. OAI-AdsBot checks pages submitted as advertising landing pages. Their names are useful because the website decisions associated with them are different. [1]

For a website owner, the practical question is which activity you want to permit and whether your website actually permits it. A publisher might welcome search discovery while restricting training collection. An advertiser may need a submitted landing page to be accessible for review. A developer investigating a failed user-requested visit needs to understand more than the site's search policy.

This guide starts with those decisions, then explains identification, robots.txt examples, verification, and troubleshooting. Technical references were reviewed on 16 November 2026. Check the linked operator documentation before making infrastructure changes.

OpenAI bots at a glance

IdentityDocumented jobMain website decisionImportant limit
OAI-SearchBotDiscovery for ChatGPT searchWhether to allow its automatic search crawlAllowing access does not guarantee selection in an answer.
GPTBotCollection that may be used for foundation-model trainingWhether to allow this training-related collectionA request does not prove a page was used in a particular model.
ChatGPT-UserCertain user-initiated ChatGPT and Custom GPT actionsWhether the requested content is accessibleOpenAI says robots.txt may not apply to user-initiated actions.
OAI-AdsBotSafety validation and relevance assessment of submitted ad pagesWhether the submitted landing page can be checkedOpenAI says it visits submitted ad pages and does not use its collected data for foundation-model training.

The roles and limitations above follow OpenAI's crawler reference. All four are request identities, rather than policy-only tokens such as Google-Extended. Ansvra's current analyzer registry recognizes all four declared identities. [1]

How a website request relates to an AI answer

A crawl is an automated request for a resource. Search discovery concerns finding content that could support a later search result. Retrieval means obtaining content for a task. Training changes a model through a separate development process. An answer citing your page is another observation, and a person following that link is another event again.

These distinctions stop a common reporting mistake: treating a crawler request as a completed journey from discovery to citation to customer. A successful request establishes that a response was served. To know whether a particular answer linked to your page, inspect the answer. To know whether someone visited, inspect the available website-traffic evidence.

Four OpenAI request purposes

Explore each documented purpose. The branches represent uses, not four required separate network requests.

  1. Your website

    The same public website can receive requests associated with different documented purposes. [1]

    Next: OAI-SearchBot, GPTBot, ChatGPT-User, OAI-AdsBot

  2. OAI-SearchBot

    Discovers content for ChatGPT search; access does not guarantee an answer citation. [1]

  3. GPTBot

    Collects content that may be used for foundation-model training; independent preference from search. [1]

  4. ChatGPT-User

    Visits for certain user actions; robots.txt may not apply. [1]

  5. OAI-AdsBot

    Checks submitted advertising pages for safety and possible relevance; not foundation-model training. [1]

Conceptual purpose map, not internal architecture or observed request data. OpenAI may reuse a crawl for search and training when both are permitted. Signed Cloud browser traffic has separate verification. [1] [4]

The diagram describes documented purposes, not a reconstruction of OpenAI's internal systems. OpenAI says that when a website allows both OAI-SearchBot and GPTBot, it may reuse one crawl's results for both purposes to avoid duplicate crawling. Separate preferences therefore do not require two visible requests. [1]

OAI-SearchBot is the identity to investigate when your question concerns automatic crawling for ChatGPT search. OpenAI says sites opting out will not appear in ChatGPT search answers, although they may still appear as navigational links. A restriction is therefore more precise than a promise that your domain will never be linked anywhere. [1]

The robots.txt token is OAI-SearchBot. OpenAI publishes its current address ranges at searchbot.json. Its reference includes a browser-like user-agent example containing OAI-SearchBot/1.4; the version can change. Match the documented identity without making one example's browser version a permanent requirement. [1]

OpenAI also documents an optional robots.txt marker in the user-agent string when fetching that file. It helps distinguish a policy lookup from another resource request, especially in incomplete logs. It is not another crawler identity. Where paths are available, inspect the actual requested path as well. [1]

If search discovery is your goal, review this bot's rules and network access together. OpenAI recommends permitting its published ranges alongside robots access. After a robots.txt update, its search systems can take approximately 24 hours to adjust. That is a documented adjustment period, not a guarantee of a new crawl, indexing, or citation within a day. [1]

GPTBot: collection that may support training

GPTBot's documented role is collecting content that may be used to train OpenAI's generative AI foundation models. The word “may” matters: seeing the bot request a page does not identify the eventual dataset, model version, or processing decision. [1]

Its robots.txt token is GPTBot, and its published address feed is gptbot.json. The reference's example contains GPTBot/1.4. As with search crawling, a version or an additional robots marker should not be mistaken for a new purpose. [1]

OpenAI describes disallowing GPTBot as indicating that a site's content should not be used for foundation-model training. This setting is independent of OAI-SearchBot. A site owner can express a training restriction while retaining search discovery. [1]

A current access decision should not be reported as proof of deletion from earlier datasets or removal from an already trained model. The crawler reference does not provide that assurance. Keep your documented policy, the date it changed, and any later request evidence separate from claims about historical content use.

ChatGPT-User: retrieval prompted by a user

ChatGPT-User is used for certain actions requested through ChatGPT and Custom GPTs. A user can ask for help that results in a page visit. OpenAI explicitly distinguishes this activity from automatic web crawling and says robots.txt rules may not apply because the action is user-initiated. [1]

The identity appears as ChatGPT-User, with a documented example containing ChatGPT-User/1.0. OpenAI publishes chatgpt-user.json for its address ranges. A request declaring the name still needs corroboration before you grant it special network treatment. [1]

Blocking ChatGPT-User in robots.txt is not the documented way to opt out of ChatGPT search. OpenAI directs website owners to OAI-SearchBot for that decision. Equally, changing a search preference is not a substitute for authentication when a page is intended to remain private. [1] [2]

Do not assume every ChatGPT browsing session will declare this identity. OpenAI separately documents signed Cloud browser traffic, discussed below. A product-level name in an answer or browser interface does not establish which request identity reached your site. [4]

OAI-AdsBot: submitted advertising landing pages

OAI-AdsBot checks the safety of web pages submitted as ads on ChatGPT. OpenAI says landing-page content may also help assess when an ad is relevant. The crawler reference limits its visits to submitted ad pages and says the collected data is not used to train generative AI foundation models. [1]

The token is OAI-AdsBot, with the documented example OAI-AdsBot/1.0. Its current address feed is adsbot.json. These details distinguish advertising validation from an ordinary search crawl or training-related request. [1]

For advertisers, OpenAI says OAI-AdsBot access is required for landing-page validation and recommends also allowing OAI-SearchBot. Its checklist covers the full submitted URL, redirects, required assets, and protection layers. A homepage loading in your browser does not show that the actual ad destination works for the bot. [3]

Have your website team inspect the complete redirect chain, including any tracking destination or separate asset host. Check that the final response contains the advertised product or service rather than a challenge, login, location error, or unrelated destination. Successful access supports the check; it is not a promise of ad approval. [3]

Choose the access policy that matches your goal

Start with a written decision for each purpose. For example: permit public search discovery; decline GPTBot collection; retain authentication on private content; permit advertising checks only where the advertising workflow needs them. The resulting configuration should reflect those decisions without weakening unrelated protections.

Allow search discovery while restricting GPTBot

This robots.txt fragment permits OAI-SearchBot at the site root while disallowing GPTBot. It implements the independent controls described by OpenAI. Merge it with your existing rules after reviewing them; it is not a complete replacement file. [1]

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

The example does not authenticate visitors, remove existing content, or change a separate firewall policy. It also does not establish controls for ChatGPT-User or signed browser sessions. Treat those as different access questions.

Restrict a section while retaining public search access

Under the Robots Exclusion Protocol, the more specific matching path rule applies. This example disallows OAI-SearchBot on paths beginning /members/ while permitting the rest. An equally specific allow/disallow conflict favors the allow rule. Review every matching group before testing your intended paths. [2]

User-agent: OAI-SearchBot
Allow: /
Disallow: /members/

A specific bot group does not automatically inherit the wildcard group's restrictions. If your general rules exclude other paths, include the appropriate restrictions in this bot's group too. Rules from multiple groups matching the same token are combined under the standard. This is why appending an isolated “allow everything” fragment can have unintended effects. [2]

Permit advertising checks where appropriate

If your policy permits OAI-AdsBot across the host, the group below expresses that preference. Check existing restrictions and required assets before applying it. Advertisers should follow OpenAI's current landing-page checklist rather than assume robots permission alone completes the setup. [3]

User-agent: OAI-AdsBot
Allow: /

Publish robots.txt at the root of the relevant origin. A rule on your main website does not configure another subdomain or a separate image host. The standard scopes the file to the scheme and authority serving it. Preserve real login and authorization controls for private resources: robots.txt is public and is not an access-security mechanism. [2]

Technical verification: identify the requester before granting access

A User-Agent header is a declaration that a requester can copy. A convincing bot name, an OpenAI documentation link inside the header, or a successful response does not independently verify ownership. Use the current official feed corresponding to the declared identity as corroborating evidence, and follow your protection provider's supported verification process. [1]

  1. Keep the evidence scoped. Record the time, hostname, path, response status, and declared identity for the request you are investigating. Use authorized infrastructure logs for any source-address investigation.
  2. Determine the real network source. At a proxied origin, a raw client-address field may identify your proxy. Use your trusted infrastructure's documented client-address handling, rather than accepting an arbitrary forwarded header.
  3. Check the correct current feed. Compare against the appropriate OpenAI published ranges, including the relevant address family. A range match supports network attribution; it does not prove later use in an answer or model.
  4. Keep changes narrow. Apply only the exception needed for the verified traffic and required resource. Retain protections for unrelated requests and review the result.

Do not copy today's feed into a permanent article allowlist. Link to the maintained source and give any operational synchronization process an owner. The crawler overview does not establish one universal crawl frequency, JavaScript capability, or crawl-delay behavior for all four identities; avoid building expectations around undocumented behavior. [1]

Signed Cloud browser traffic is a separate case

OpenAI's Cloud browser guidance describes HTTP Message Signatures: requests include Signature, Signature-Input, and a Signature-Agent value of "https://chatgpt.com". Verification uses the published signing keys and the signature standard. Merely seeing those headers is not successful verification. [4]

Use your provider's supported verified-agent integration or implement the documented signature verification. Preserve the necessary headers through proxies. OpenAI documents provider-specific agent labels, which can differ from user-agent tokens; do not turn a dashboard label into a guessed robots.txt rule. This signed-browser mechanism does not mean every GPTBot or search request carries such a signature. [4]

Troubleshoot access without confusing it with visibility

SymptomCheck firstWhat to do next
No recorded requestsWebsite, date range, observation coverage, and source deliveryEstablish that requests could have been observed before changing website policy.
403 despite an allow ruleWhich layer denied the exact request: edge protection, origin, login, or a challengeInvestigate the verified requester and resource; a robots rule does not override network enforcement.
429 responsesRate-limit evidence, timing, and the responsible ruleAdjust a justified limit narrowly and retest; do not invent a crawler's supported delay setting.
200 but unusable contentReturned body, redirects, and required assetsConfirm the response contains the intended content rather than a challenge or empty shell.
A policy change seems ineffectiveCorrect host, matching groups, path, and current fileCorrect the rule and allow the documented search adjustment period; do not promise an immediate new crawl.
Crawls but no citationsActual saved answers and their linked URLsAssess answer observations separately; successful access does not guarantee selection.

OpenAI's advertiser guidance specifically identifies CDN and firewall restrictions, JavaScript challenges, CAPTCHAs, authentication, and geographic rules as potential access problems. These are useful places to investigate when the evidence points to a failed landing-page check. They are not reasons to disable protection across an entire website. [3]

A test request that copies a bot's user-agent text can reveal how your server responds to that text. It cannot reproduce the operator's network address, verification state, rendering behavior, or internal processing. Label the result as a response test and confirm actual subsequent activity separately.

Review recorded activity in Ansvra

When AI Agent Analyzer is available for your account and has request records for the website, use it to investigate the declared bot, requested pages, dates, and recorded response information. Ansvra recognizes OAI-SearchBot, GPTBot, ChatGPT-User, and OAI-AdsBot. Identity matching is separate from verifying that OpenAI operated the requester.

Start with the correct project and time period, then compare the relevant bots and requested pages. Review the source's coverage before interpreting an empty report. Uploaded history represents saved activity; it does not establish an ongoing connection. A missing observation should remain unknown rather than becoming evidence that the bot never visited.

Use the AI Agent Analyzer setup guide for current connection and access instructions. Review human visits separately in Traffic from AI, and inspect source links in saved AI answer evidence. These observations can inform the same investigation without proving a measured sequence between them.

Frequently asked questions

Can I allow ChatGPT search while blocking training collection?

Yes. OpenAI documents independent OAI-SearchBot and GPTBot preferences. Review the example above and merge the intended rules with your existing policy. Allowing search crawling still does not guarantee a search appearance. [1]

Does blocking GPTBot stop all ChatGPT visits?

No. GPTBot addresses training-related collection. Search discovery, certain user-requested visits, advertising checks, and signed browser traffic have separately documented roles. Choose the relevant control for the activity you are investigating. [1] [4]

Does a ChatGPT-User request mean my page appeared in search?

No. OpenAI says ChatGPT-User serves certain user actions and does not determine search inclusion. Inspect actual answer evidence for a citation and use OAI-SearchBot for the documented search opt-out. [1]

Why is OAI-AdsBot different from GPTBot?

It checks submitted ad pages for safety and potential relevance. OpenAI says the data it collects is not used for foundation-model training. Advertisers should follow its dedicated access checklist. [1] [3]

Should I use noindex, noai, or llms.txt as a universal OpenAI control?

Do not assume these mechanisms are interchangeable. The examples here implement OpenAI's documented bot-specific preferences. A page directive and a crawl restriction are different mechanisms, and an uncrawled page cannot reliably deliver a new directive through its body. Verify current operator support before relying on an additional mechanism. [1] [2]

How long should I wait after changing robots.txt?

OpenAI gives approximately 24 hours for its search systems to adjust. It does not describe that period as a guaranteed crawl or citation deadline, and the statement should not be generalized to every bot or request type. [1]

Can a robots rule protect confidential pages?

No. The Robots Exclusion Protocol is not access authorization. Use authentication and authorization for confidential information; keep public crawl preferences as a separate layer. [2]

Summary

Choose a policy for each purpose, review the exact bot and host, and verify the relevant request evidence before changing access. Search discovery, potential training collection, user-requested retrieval, and advertising checks need different interpretations. Once access is understood, review citations and human visits with their own evidence.

Sources

  1. OpenAI: Overview of OpenAI CrawlersAccessed
  2. IETF: RFC 9309, Robots Exclusion ProtocolAccessed
  3. OpenAI: Advertiser Guidance for Allowing OpenAI Web CrawlersAccessed
  4. OpenAI: ChatGPT Work’s Cloud browser allowlistingAccessed

Discussion

Join the discussion

Start the conversation

Share a question or tell us what worked for you.

Comments appear after review. How we handle comments

The Ansvra journal

Keep reading

Browse all articles

Ansvra

A clearer next step.

MEET YOUR AI GUIDE

A little clarity.
A clearer next move.

Ask about Ansvra, AI visibility, or your next step with content.

Opening chat…