Quick Answer: ClaudeBot is Anthropic's web crawler that collects public pages for training its Claude models. It is one of three Anthropic bots: Claude-User fetches pages when a person asks Claude a question, and Claude-SearchBot indexes pages for Claude's search results. Blocking ClaudeBot alone does not remove you from Claude's answers.
If ClaudeBot has appeared in your server logs, you are looking at the training crawler, not the one that decides whether Claude cites you. Anthropic documents three separate robots, each with its own robots.txt switch, in its crawler policy of 7 April 2026. Most of the confusion about "blocking Claude" comes from treating the three as one.
What does ClaudeBot do on your site
ClaudeBot fetches public web content that may contribute to training Anthropic's generative models. Anthropic's own description is that it "helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training."
It behaves like a conventional crawler: it reads robots.txt first, honours Disallow directives, and supports the non-standard Crawl-delay extension. Anthropic states that its bots respect anti-circumvention measures such as CAPTCHAs, so a page behind one is simply not collected.
At a glance
| Bot | Job | What blocking it does |
|---|---|---|
| ClaudeBot | Collects public content for model training | Signals that future content should be excluded from training data |
| Claude-User | Fetches a page when a Claude user asks a question | Claude cannot retrieve your page for that user, reducing visibility in user-directed search |
| Claude-SearchBot | Indexes content to improve Claude's search results | Your pages drop out of Claude's search index, reducing visibility and accuracy |
How to recognise it in logs
Look for the token ClaudeBot in the user-agent string, and confirm the source IP against Anthropic's published list at claude.com/crawling/bots.json. A request carrying the name from an address that is not on that list is a spoof, and spoofed AI user agents are common enough that the IP check is the only reliable test.
ClaudeBot vs Claude-User vs Claude-SearchBot
ClaudeBot trains models, Claude-User serves a live question, and Claude-SearchBot builds the search index; only the second and third affect whether Claude cites you. Anthropic's doc states that disabling Claude-User "may reduce your site's visibility for user-directed web search" and disabling Claude-SearchBot "may reduce your site's visibility and accuracy in user search results."
That split mirrors the pattern OpenAI uses, where GPTBot handles training and OAI-SearchBot handles search. The robots.txt directives for each AI bot differ, so a rule written for one does nothing to the others.
Which one matters for AI visibility
Claude-User is the bot that fetches your page at the moment a user's question could be answered by it, which is the moment a citation is earned. Claude-SearchBot decides whether your page is in the index that Claude searches at all. ClaudeBot's crawl affects what a future model has memorised, which is a slower and far less controllable effect.
Why the names get confused
Log analysers often group all three under "Anthropic" or "Claude", and older articles were written before Claude-User and Claude-SearchBot existed. When someone says they "blocked ClaudeBot" and then lost Claude citations, they almost always blocked all three with a wildcard rule.
Should you block ClaudeBot
Block ClaudeBot only if you have decided your content should not be used for model training, and leave Claude-User and Claude-SearchBot allowed if you want Claude to cite you. Those are two separate decisions, and the robots.txt syntax lets you make them separately.
The case for allowing it
Training data shapes what a model believes about your category before it ever searches. A brand whose product pages, specifications and comparisons are in the training set is more likely to be named from memory on questions where Claude does not search at all. Entity signals that AI models learn are built partly from exactly this crawl.
The case for blocking it
Publishers who license content, or who object to training use on principle, have a clear reason to disallow ClaudeBot. Anthropic's doc says blocking it "signals that the site's future materials should be excluded" from training datasets, which is a forward-looking opt-out rather than a retroactive one.
The mistake to avoid
Do not block by IP range as your only method. Anthropic warns that IP blocking "may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file." The robots.txt directive is the mechanism the crawler is built to obey.
How to write the robots.txt rules
Write one rule block per bot, because each Anthropic bot reads only the block addressed to it. The examples below are Anthropic's own syntax.
To slow ClaudeBot without blocking it:
User-agent: ClaudeBot
Crawl-delay: 1
To block training while keeping search and user fetches open:
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
Subdomains and staging
Anthropic asks that you repeat the rules on every subdomain you want opted out, since robots.txt is read per host. A staging subdomain with no robots.txt is crawlable by default, which is how draft content ends up in a training set.
What a CDN firewall does
A WAF rule that challenges every unknown bot will also challenge Claude-User, and Anthropic's bots do not solve CAPTCHAs. If your CDN has a managed "AI bots" toggle, check whether it lumps the search and user agents in with the training crawler before you switch it on; many do.
How to check whether Claude can still reach your pages
Ask Claude a question that your page answers, with web search enabled, and see whether it fetches and cites the page. That is a direct test of Claude-User; a page that is fetched but not cited has a content problem, and a page that is never fetched has an access problem.
Reading your own logs
Filter access logs for Claude-User and Claude-SearchBot over 30 days and note the status codes. A run of 403s means your firewall is refusing them, and no requests at all usually means the index never picked the pages up. A run of 200s on pages that never get cited means the answer is not extractable, which is a writing problem rather than an access one.
Keep the two diagnoses apart in your notes. Access problems are fixed in a config file in an afternoon; extraction problems are fixed one page at a time, and measuring which pages get read but not cited tells you where to start.
Where this fits in a wider audit
Crawler access is one line in an AI visibility audit, and it is the cheapest one to fix. It is also the line that the rest of answer engine optimization depends on, since a page no bot can fetch cannot be quoted.
Once the three Anthropic bots are configured deliberately, repeat the exercise for the OpenAI, Perplexity and Google crawlers. Each of them splits training from search in its own way, and each needs its own block.