ClaudeBot User Agent Strings Plus IP Verification in 2026

Quick Answer: ClaudeBot is Anthropic's web crawler that collects public pages for training its Claude models. It is one of three Anthropic bots: Claude-User fetches pages when a person asks Claude a question, and Claude-SearchBot indexes pages for Claude's search results. Blocking ClaudeBot alone does not remove you from Claude's answers.

If ClaudeBot has appeared in your server logs, you are looking at the training crawler, not the one that decides whether Claude cites you. Anthropic documents three separate robots, each with its own robots.txt switch, in its crawler policy of 7 April 2026. Most of the confusion about "blocking Claude" comes from treating the three as one.

What does ClaudeBot do on your site

ClaudeBot fetches public web content that may contribute to training Anthropic's generative models. Anthropic's own description is that it "helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training."

It behaves like a conventional crawler: it reads robots.txt first, honours Disallow directives, and supports the non-standard Crawl-delay extension. Anthropic states that its bots respect anti-circumvention measures such as CAPTCHAs, so a page behind one is simply not collected.

At a glance

How to recognise it in logs

Look for the token ClaudeBot in the user-agent string, then confirm the source IP against Anthropic's published list. The two sections below give the exact strings and the verification steps.

What is the ClaudeBot user agent string

ClaudeBot's robots.txt token is ClaudeBot, and its full user-agent string in server logs is Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; [email protected]). Anthropic's crawler policy documents the three tokens; the full strings below are the ones logged in the wild, as compiled by Search Engine Journal's AI crawler list.

Match on the token, not the whole string. Anthropic does not promise the surrounding text stays fixed, and older ClaudeBot requests have carried a different contact address, so a filter that expects the exact string will miss real visits.

Older Anthropic user agents you may still see

Before ClaudeBot, Anthropic crawled under other names, and some robots.txt files still block anthropic-ai or Claude-Web. Those lines are harmless, but they do nothing to the three current bots, so add explicit blocks for the tokens in the table if blocking is what you intend.

How to verify ClaudeBot by IP address

Verify ClaudeBot by checking the request's source IP against Anthropic's list at claude.com/crawling/bots.json. Anthropic's crawler policy says a crawler whose source IP is on that list is coming from Anthropic.

When fetched on 5 October 2026, the file was dated 2 October 2026 and held 28 IPv4 prefixes, the largest being 216.73.216.0/22. The list changes, so fetch it on a schedule rather than copying today's ranges into a firewall rule.

Verification in four steps

1. Pull the requests. Extract every log line whose user agent contains ClaudeBot, Claude-User or Claude-SearchBot.
2. Fetch the current list. Download bots.json and read every ipv4Prefix value.
3. Match each IP. Check whether the request's source address falls inside any listed prefix.
4. Treat misses as spoofs. A request that carries an Anthropic name from an unlisted address is not Anthropic's crawler; handle it like any other unknown scraper.

import ipaddress, json, urllib.request
prefixes = [ipaddress.ip_network(p["ipv4Prefix"]) for p in
            json.load(urllib.request.urlopen("https://claude.com/crawling/bots.json"))["prefixes"]]
def is_anthropic(ip):
    return any(ipaddress.ip_address(ip) in net for net in prefixes)

How to detect ClaudeBot in server logs

Count ClaudeBot requests per day with a case-insensitive search on the access log, then split them by status code. On a standard Nginx or Apache combined log, these two commands do it:

grep -i "claudebot" access.log | awk '{print $4}' | cut -d: -f1 | sort | uniq -c
grep -i -E "claude-user|claude-searchbot" access.log | awk '{print $9}' | sort | uniq -c

The first line shows how often the training crawler visits. The second shows whether the two bots that decide citations get 200s or are being refused with 403s by a firewall.

ClaudeBot vs Claude-User vs Claude-SearchBot

ClaudeBot trains models, Claude-User serves a live question, and Claude-SearchBot builds the search index; only the second and third affect whether Claude cites you. Anthropic's doc states that disabling Claude-User "may reduce your site's visibility for user-directed web search" and disabling Claude-SearchBot "may reduce your site's visibility and accuracy in user search results."

That split mirrors the pattern OpenAI uses, where GPTBot handles training and OAI-SearchBot handles search. The robots.txt directives for each AI bot differ, so a rule written for one does nothing to the others.

Which one matters for AI visibility

Claude-User is the bot that fetches your page at the moment a user's question could be answered by it, which is the moment a citation is earned. Claude-SearchBot decides whether your page is in the index that Claude searches at all. ClaudeBot's crawl affects what a future model has memorised, which is a slower and far less controllable effect.

Why the names get confused

Log analysers often group all three under "Anthropic" or "Claude", and older articles were written before Claude-User and Claude-SearchBot existed. When someone says they "blocked ClaudeBot" and then lost Claude citations, they almost always blocked all three with a wildcard rule.

Should you block ClaudeBot

Block ClaudeBot only if you have decided your content should not be used for model training, and leave Claude-User and Claude-SearchBot allowed if you want Claude to cite you. Those are two separate decisions, and the robots.txt syntax lets you make them separately.

The case for allowing it

Training data shapes what a model believes about your category before it ever searches. A brand whose product pages, specifications and comparisons are in the training set is more likely to be named from memory on questions where Claude does not search at all. Entity signals that AI models learn are built partly from exactly this crawl.

The case for blocking it

Publishers who license content, or who object to training use on principle, have a clear reason to disallow ClaudeBot. Anthropic's doc says blocking it "signals that the site's future materials should be excluded" from training datasets, which is a forward-looking opt-out rather than a retroactive one.

The mistake to avoid

Do not block by IP range as your only method. Anthropic warns that IP blocking "may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file." The robots.txt directive is the mechanism the crawler is built to obey.

How to write the robots.txt rules

Write one rule block per bot, because each Anthropic bot reads only the block addressed to it. The examples below are Anthropic's own syntax.

To slow ClaudeBot without blocking it:

User-agent: ClaudeBot
Crawl-delay: 1

To block training while keeping search and user fetches open:

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

Subdomains and staging

Anthropic asks that you repeat the rules on every subdomain you want opted out, since robots.txt is read per host. A staging subdomain with no robots.txt is crawlable by default, which is how draft content ends up in a training set.

What a CDN firewall does

A WAF rule that challenges every unknown bot will also challenge Claude-User, and Anthropic's bots do not solve CAPTCHAs. If your CDN has a managed "AI bots" toggle, check whether it lumps the search and user agents in with the training crawler before you switch it on; many do.

How to check whether Claude can still reach your pages

Ask Claude a question that your page answers, with web search enabled, and see whether it fetches and cites the page. That is a direct test of Claude-User; a page that is fetched but not cited has a content problem, and a page that is never fetched has an access problem.

Reading your own logs

Filter access logs for Claude-User and Claude-SearchBot over 30 days and note the status codes. A run of 403s means your firewall is refusing them, and no requests at all usually means the index never picked the pages up. A run of 200s on pages that never get cited means the answer is not extractable, which is a writing problem rather than an access one.

Keep the two diagnoses apart in your notes. Access problems are fixed in a config file in an afternoon; extraction problems are fixed one page at a time, and measuring which pages get read but not cited tells you where to start.

Where this fits in a wider audit

Crawler access is one line in an AI visibility audit, and it is the cheapest one to fix. It is also the line that the rest of answer engine optimization depends on, since a page no bot can fetch cannot be quoted.

Once the three Anthropic bots are configured deliberately, repeat the exercise for the OpenAI, Perplexity and Google crawlers. Each of them splits training from search in its own way, and each needs its own block.