Key takeaways
- ClaudeBot collects material that may contribute to training, Claude-SearchBot supports search quality, and Claude-User fetches content for user requests. Anthropic documents separate preferences for these activities. [1]
- Anthropic says its bots honor robots.txt, including controls on Claude-User. Do not transfer another platform's user-request exceptions to Claude. [1]
- Current guidance links to published crawler IP ranges. Hosted browser traffic uses the separate Claude-Agent identity and its own published address listing. [1] [3]
- Training-use preferences, crawl restrictions, search visibility, citations, and customer visits need separate evidence. None proves the next step occurred.
Anthropic uses three named web robots for different purposes: ClaudeBot, Claude-SearchBot, and Claude-User. Understanding which one requested a page helps you choose the right website policy. A training-related crawler, a search crawler, and a user-directed fetcher are doing different jobs even when they ask for the same URL. [1]
Anthropic also documents Claude-Agent for a browser hosted on its infrastructure and driven by a user-directed agent. This is a separate category of access, not another name for Claude-User. A website owner investigating browser activity should examine that identity independently. [3]
This guide explains the differences in plain language before covering robots.txt examples, training preferences, request verification, and troubleshooting. Technical references were reviewed on 17 November 2026. Because the documentation and emerging preference mechanisms change, recheck the linked sources before applying a website-wide access change.
Which Claude bot is visiting your website?
| Identity | Documented purpose | Website-owner decision | Current Ansvra coverage |
|---|---|---|---|
| ClaudeBot | Collects web content that could contribute to model training | Whether to permit training-related collection | Recognized declared request identity |
| Claude-SearchBot | Analyzes content to improve search relevance and accuracy | Whether to permit this search-related crawling | Recognized declared request identity |
| Claude-User | Retrieves website content in response to user requests | Whether to permit those user-directed retrievals | Recognized declared request identity |
| Claude-Agent | Identifies Anthropic-hosted agentic browser use | Whether to allow that separate browser traffic | Not currently a registered analyzer identity |
Anthropic's crawler reference documents the first three roles; its browser reference documents the fourth. Ansvra's recognition coverage is a separate product fact. A recognized name does not by itself verify that Anthropic sent a particular request. [1] [3]
How crawling, fetching, and training differ
Think of the distinction as three different reasons to consult a library. One visitor gathers material for future learning, another organizes useful references, and another retrieves a particular item because someone asked for it. The analogy explains the purposes, but a request log does not reveal the full later journey of the material.
A crawl is an automated request for a resource. Retrieval obtains material for a task. Training changes a model's learned behavior through a separate development process. Grounding supplies relevant information when generating an answer. Finding a page, fetching it, training on it, citing it, and sending someone to it are distinct events.
Select an identity to inspect its documented role. Browser activity is separate from the three crawler and fetcher identities.
- Your public website
Different purposes can involve the same public page. [1]
Next: ClaudeBot, Claude-SearchBot, Claude-User, Claude-Agent
- ClaudeBot
Collects web material that could contribute to model training; future collection preferences are separate from past use. [1]
- Claude-SearchBot
Analyzes content to improve search relevance and accuracy. Allowed crawling does not guarantee an answer citation. [1]
- Claude-User
Retrieves content at user direction. Anthropic documents website-owner controls on this identity. [1]
- Claude-Agent
Separate hosted browser identity; use the browser-specific address listing and access guidance. [3]
Conceptual purpose map, not internal architecture or measured traffic. Training-use preferences and crawler access are different decisions. Ansvra currently recognizes the three crawler/fetcher names; Claude-Agent is outside its registered identities. [1] [3]
This purpose map is conceptual, not a claim about Anthropic's internal architecture or a measured sequence of requests. The crawler reference identifies roles; it does not expose every processing step, every search-source relationship, or the eventual destination of each fetched page. [1]
ClaudeBot: collection that may contribute to training
ClaudeBot is Anthropic's training-related crawler. Anthropic describes its collection as material that could contribute to the utility and safety of its generative models. Restricting ClaudeBot signals that the site's future material should be excluded from its training datasets. Preserve that forward-looking scope when explaining the effect of a block. [1]
The exact robots.txt token is ClaudeBot. Use that name when expressing its crawl preference, rather than a model name or a guessed company-wide token. The current reference supplies this token but does not provide a complete browser-style user-agent example for each of the three robots. There is no reason to invent a versioned header to configure the documented rule. [1]
A logged ClaudeBot request is evidence of a declared identity and a response to a resource request. Even after corroborating its source, it does not prove inclusion in a particular dataset or training run. It also does not identify a model version, a retention period, or whether your text was later reproduced.
If your policy is to restrict training-related collection while retaining search and user-requested access, name ClaudeBot explicitly and configure the other purposes independently. Do not describe that change as deleting previously collected content or reversing earlier training: the reviewed crawler guidance offers no such assurance. [1]
Claude-SearchBot: content for search quality
Claude-SearchBot explores and analyzes online content to improve search response relevance and accuracy. Anthropic says disabling it prevents its system from indexing your content for search optimization and may reduce your site's visibility and accuracy in search results. This is an access and discovery consequence, not a guaranteed ranking change or a promise of citation when access is allowed. [1]
The exact token is Claude-SearchBot. Decide whether public pages should be available to that bot, then check that your network and the returned content are consistent with that decision. A permissive robots group cannot overcome a login requirement, a firewall denial, or a challenge page.
Claude's product documentation separately explains that web search can consult multiple sources and provide citations in its response. This helps explain why website owners care about search access, but it does not prove that every source link corresponds to a fresh Claude-SearchBot request on their server. Inspect the actual answer when the question concerns a citation. [4]
Search access and training preferences should therefore be reviewed separately. A publisher can choose to welcome search-related crawling while restricting ClaudeBot. Keep the resulting rules readable enough that a later reviewer can understand which purpose each group controls.
Claude-User: retrieval in response to a user
Claude-User supports user-directed requests. When someone asks Claude a question, the service may retrieve a relevant website through this identity. Anthropic says disabling Claude-User prevents its system from retrieving the site's content in response to a user query, potentially reducing visibility in user-directed web search. [1]
The robots.txt token is Claude-User. Its user-directed role does not make it exempt from the preferences described in Anthropic's documentation. In particular, do not copy another operator's statement that robots rules may not apply to user actions and present it as Claude's behavior. This platform documents controls on this fetcher. [1]
Web fetch is also described in Claude's product guidance: a user can provide a specific URL for Claude to retrieve and discuss. The feature description is useful context; the crawler's identity and access behavior remain governed by the specific crawler reference. It is not evidence that every browser, tool, or third-party application using Claude sends the same request identity. [4]
For troubleshooting, ask whether the user needed a specific page that your policy deliberately restricts. An intentional block can explain failed retrieval without indicating that the website is broken. If access was intended, investigate the exact URL, response, redirect chain, and applicable host policy.
Claude-Agent: Anthropic-hosted browsing is separate
Anthropic-hosted browsers can complete tasks at a user's direction. Their requests use the browser's normal header with an added Claude-Agent token. The official example includes Claude-Agent/1.0, but other parts of the browser string can change. Match the identity token instead of freezing an entire example header. [3]
Anthropic publishes browser addresses through its agent address listing, under Claude-Agent. Its guidance recommends keeping rules updated from that file and checking the source address because anyone can copy the User-Agent header. It describes the publication format as evolving rather than an established standard. [3]
Browser access preferences and the crawler/fetcher preferences are independent. Do not treat a Claude-User rule as a complete policy for every hosted browser action, or turn a browser identity into a made-up search control. Follow the browser-specific guidance for your infrastructure. [3]
Ansvra's current bot registry does not include Claude-Agent. This guide covers the operator's documented identity, but that coverage does not add detection to the product. Browser traffic can therefore require a separate investigation in your authorized website infrastructure records.
Manage each purpose independently in robots.txt
First write down your intended policy for each activity. Then review the existing file before merging any examples. The snippets below demonstrate specific outcomes; none is a complete replacement for all of your website's existing rules.
Permit search and user retrieval while restricting ClaudeBot
This example disallows ClaudeBot while explicitly permitting the two other documented robots. It expresses a choice about Anthropic's purposes; it does not configure other operators, guarantee a citation, or authorize access to private content. [1]
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
Before using root-wide allow rules, bring across any path restrictions that should apply to these bots. Under the Robots Exclusion Protocol, a matching named group does not automatically inherit restrictions from a wildcard group. Separate matching groups for the same token are combined, so appending a contradictory group is not a reliable override. [2]
Keep a specific section out of user-directed retrieval
To permit Claude-User on public paths while excluding a section, make the restriction explicit. Here, the more specific /members/ rule takes precedence over the root allow. This is crawl guidance; authentication remains necessary if the section contains private information. [1] [2]
User-agent: Claude-User
Allow: /
Disallow: /members/
Use case-sensitive path matching when reviewing URLs. Test both the intended blocked path and a neighboring allowed path. Equivalent allow and disallow matches favor allow under the standard, so review exceptions rather than assuming that the last line wins. [2]
Restrict all three documented robots
If that is your publishing policy, name each of the three robots. This expresses the relevant crawl restrictions without inventing a universal Anthropic token. It does not establish a complete access policy for Claude-Agent browsers or remove content from every other source. [1] [3]
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: Claude-User
Disallow: /
Put robots.txt at the relevant origin's root. Anthropic specifically says to repeat the opt-out for each subdomain you want covered. A policy at your main hostname does not configure an independently hosted publication or asset domain. Keep the file reachable so a compliant crawler can read your preferences. [1] [2]
Limit crawling activity with Crawl-delay
Anthropic explicitly documents support for the non-standard Crawl-delay extension and provides the following ClaudeBot example. It is an operator-supported extension, not a universal rule that every crawler implements. [1]
User-agent: ClaudeBot
Crawl-delay: 1
Use this when your intended policy permits collection but seeks to limit its activity. It is not a substitute for Disallow when you want exclusion. The reference does not establish a guaranteed request volume across all infrastructure or a new-crawl deadline after changes. Observe subsequent requests and investigate unexpected load with their timestamps and response outcomes.
The newer training preference: what is changing?
Anthropic says it began recognizing a general machine-readable training preference on 30 October 2026 and is rolling it out across services over time. The linked specification is an individual Internet-Draft, not a finalized IETF standard. Keep the rollout caveat attached to any implementation advice. [1] [5]
The draft uses the policy token __usage_ai-training inside ordinary robots.txt groups. A restrictive preference can be expressed as follows. This is a content-use preference, not the name of an extra crawler you should expect to see in request history. [5]
User-agent: __usage_ai-training
Disallow: /
Anthropic explains its precedence separately: explicit ClaudeBot rules are treated as more specific than the general signal. For a path, it follows a rule naming ClaudeBot even if the general setting differs; when ClaudeBot is not mentioned, it follows the general setting described in its guidance. Audit both before claiming that your configuration declines training use. [1]
For example, a root-wide ClaudeBot allow paired with a general training disallow expresses conflicting intentions under that precedence. If your decision is to restrict ClaudeBot, make its own group restrictive too. Do not assume the generic group automatically cancels an explicit bot rule.
The draft also distinguishes training-use preferences from network crawl controls and does not treat a wildcard group as the general training preference. Retain purpose-specific crawler groups for traffic decisions. Do not describe this signal as universally enforced by every platform or as proof that earlier collection has been deleted. [5]
Verify a request without trusting its name alone
A user-agent string is supplied by the requester. It is useful for classification but easy to copy. Current Anthropic guidance links to a published crawler address list; older advice saying that no crawler ranges are published should be rechecked against that current reference. [1] [6]
- Identify the exact request. Keep the hostname, URL path, timestamp, status, and declared identity together. A report about another host or date range can mislead the investigation.
- Establish the real source address. Behind a proxy, your origin may see the proxy's address. Use your infrastructure's trusted client-address handling; do not accept an arbitrary forwarded header as evidence.
- Use the appropriate maintained list. The crawler reference links to bots.json. Claude-Agent browser guidance links to the separate agent listing. Do not assume the two lists have the same scope or format. [1] [3]
- State the result precisely. A matching declared identity and source range support network attribution. They do not identify a model, prove training use, or establish that the requested page was cited.
Keep operational rules current with the published sources rather than embedding a permanent list in an article. If you cannot corroborate the source, report the identity as declared or identified rather than verified. Do not invent reverse-DNS suffixes or a signature mechanism for these bots.
Anthropic cautions that blocking addresses alone may not reliably express an opt-out and can prevent its bots from reading robots.txt. Use the documented file for publishing preferences while retaining the security controls your website needs. The presence of a source address on a list is not a reason to grant access to private resources. [1]
Investigate access failures and content visibility separately
| Symptom | Evidence to inspect | Next action |
|---|---|---|
| No recorded bot activity | Site, dates, source coverage, and recognition support | Establish observation coverage before treating absence as a block. |
| 403 or a challenge | Exact response and protection-layer event | Check whether access was intentionally denied; narrowly correct an unintended restriction. |
| 429 responses | Request timing and the responsible rate rule | Review allowed activity and operator-supported delay guidance without assuming fixed volumes. |
| 200 but no useful page | Actual returned body, redirects, and login state | Confirm that the response contains the intended content rather than a shell or challenge. |
| User-requested retrieval fails | Claude-User policy for the exact host and path | Decide whether the restriction is intentional before changing it. |
| Requests appear but citations do not | Saved answers and exact source links | Investigate selection separately from successful website access. |
Anthropic says its bots respect anti-circumvention technologies and will not try to bypass CAPTCHAs on crawled sites. A page that requires a challenge can therefore explain failed access. That does not justify disabling protections everywhere: first identify the bot, affected public path, and rule responsible. [1]
For content visibility, consult Anthropic's separate removal guidance. It describes measures including password protection, noindex directives for search partners, bot restrictions, and a request process for an existing URL. These mechanisms address different situations; blocking one named crawler is not a complete explanation of every source available to web search. Follow the relevant current instructions and their stated scope. [7]
A noindex instruction must be available to the system that needs to discover it. It is different from denying retrieval, and neither is an authorization system. For confidential material, use real access controls. If your concern is a specific existing output or URL, use the documented review process rather than promise that a new robots rule immediately erases it. [2] [7]
A test that copies a bot name can show how your server responds to that header. It cannot recreate Anthropic's address, task context, or processing. Treat it as a response test and inspect genuine subsequent requests separately. The reviewed references do not establish a universal rendering capability, cache duration, or policy-propagation deadline for all three bots.
Review Claude activity in Ansvra
When AI Agent Analyzer is available for your account and has records for the website, review the declared bot, requested pages, time period, and recorded responses. Its current registry recognizes ClaudeBot, Claude-SearchBot, and Claude-User. It does not currently register Claude-Agent or create a request series for the general training-policy token.
Start with the correct project and period, then inspect relevant pages and response outcomes. Check the source's coverage before interpreting an empty view. Uploaded history is saved activity, not evidence of continuing delivery. The analyzer's identity classification does not independently authenticate Anthropic or establish which model used a page.
Use the AI Agent Analyzer guide for current setup and access instructions. Review Traffic from AI separately for recorded human visits and saved answer citations for linked sources. For platform-specific differences, compare the OpenAI crawler guide; do not assume its control behavior transfers to Claude.
Frequently asked questions
Can I allow Claude search while restricting training collection?
Yes. Review Claude-SearchBot, Claude-User, and ClaudeBot separately. The example above permits the first two and disallows ClaudeBot. Also review any general training preference for consistency with Anthropic's stated precedence. Access still does not guarantee a citation. [1]
Does blocking ClaudeBot also block Claude-User?
A rule naming ClaudeBot addresses that identity. Anthropic documents Claude-User separately for user-requested retrieval. Configure each purpose and inspect all applicable groups rather than assuming that one name covers every activity. [1]
Does Claude-User honor robots.txt?
Anthropic says its bots honor those directives and specifically describes disabling Claude-User as preventing retrieval for user queries. Use its own documentation rather than another platform's user-action exception. [1]
Does Anthropic publish IP ranges?
The current crawler reference links to bots.json, and the hosted-browser reference points to a separate Claude-Agent listing. Verify against the relevant current source rather than relying on older advice or a copied permanent list. [1] [3]
Is Claude-Agent the same as Claude-User?
No. Claude-Agent marks Anthropic-hosted browser activity. Claude-User is a separately documented user-directed fetcher. Their access guidance and Ansvra recognition coverage differ. [1] [3]
Should I add older names from a third-party bot list?
Use the current operator-documented identities for new instructions. Names absent from the reviewed reference are not established as current or retired by that absence alone. Keep an unexpected declared name in the investigation record and seek corroboration before assigning it a purpose or special access.
Is the general training token a new crawler?
No. It expresses a use preference in an evolving Internet-Draft. Anthropic describes a rollout and gives explicit ClaudeBot rules priority. It is not a substitute for every crawler rule or proof of a universal industry policy. [1] [5]
Does a block remove previously collected material?
The crawler guidance describes future collection preferences. It does not establish deletion from historical datasets or reversal of training. For an existing URL or output concern, consult Anthropic's separate content-reporting and removal guidance. [1] [7]
Summary
Choose a policy for training collection, search crawling, user-directed retrieval, and hosted browsing separately. Keep explicit bot rules and the newer training preference consistent, verify the relevant request evidence, and investigate the exact failing resource. Use saved answers and website analytics for citation and visit questions that a crawler log cannot answer.
Sources
- Anthropic: crawler purposes, website preferences and training-signal rolloutAccessed
- IETF: RFC 9309, Robots Exclusion ProtocolAccessed
- Anthropic: hosted browser identification and Claude-AgentAccessed
- Anthropic: Enable and use web searchAccessed
- Internet-Draft: Machine-Readable Training Preference Signal, revision 01 (work in progress)Accessed
- Anthropic: published crawler address rangesAccessed
- Anthropic: Report, block, and remove content from ClaudeAccessed
Discussion
Join the discussion