One single line of a text file may efficiently erase your brand from ChatGPT's responses. Such is the risk you take with robots.txt in the AI age, and many websites have done it without knowing it.
These days, AI crawlers are constantly scouring your site, and robots.txt is your main method of controlling these crawlers. Blocking all of them blindly is the lethal error that can cost you your online visibility, as some of those bots are the ones that allow you to be featured in AI search engine responses.
This guide will tell you which bots matter, along with one vital distinction that changes everything and how to configure your text file properly.
What does robots.txt control, and what does it not?
Robots.txt denotes a text file located at the top of the domain and gives information to different crawlers about the accessibility of the site's content. All well-known AI companies take robots.txt into account, making it a well-known way to grant bots some access to your site.
However, one should not forget about its limitations. It is an optional guideline and not a security mechanism. A directive only functions in case the bot decides to follow it as some robots completely disregard it.
It governs crawling, not what a model already learned, and it cannot claw back content that was crawled last year. For anything you truly need to stop, robots.txt is the first layer, not the only one.
Which AI bots are worth knowing?
Every big AI company operates multiple bots which perform several tasks. Below are the bots relevant in the context of visibility and control.
| Bot | Company | What it does |
|---|---|---|
| GPTBot | OpenAI | Crawls content that may train OpenAI models |
| OAI-SearchBot | OpenAI | Indexes sites to surface in ChatGPT search answers |
| ChatGPT-User | OpenAI | Fetches a page when a user asks ChatGPT to |
| ClaudeBot | Anthropic | Crawls content that may train Anthropic models |
| Claude-SearchBot | Anthropic | Indexes sites for Claude's search results |
| Claude-User | Anthropic | Fetches a page for a user's live Claude request |
| Googlebot | Search indexing, including AI Overviews and AI Mode eligibility | |
| Google-Extended | Controls Gemini training and grounding, separate from Search | |
| PerplexityBot | Perplexity | Indexes sites for Perplexity answers |
| Bingbot | Microsoft | Bing indexing, which feeds Copilot |
There are also a number of bots, like CCBot which comes from Common Crawl, Meta's crawlers, and the Amazon, Apple and ByteDance's crawlers, engaged in crawling for the main task of collecting training data. You may block these crawlers, but it will not in any way affect your AI answers searchability. Bytespider is known for its aggressive crawling practices and for violating rules.
Training bots vs answer bots: what is the difference?
This one key principle distinguishes the clever robots.txt from the one which conceals your identity. Here are the two classifications of AI that lead to opposite implications for your visibility.
Bots, such as GPTBot, Google-Extended, and ClaudeBot, search for material that is useful for training purposes. By preventing these bots from accessing your material, you can keep your information out of future training. However, it does not prevent you from being included in AI responses right now.
Answer bots, like OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, and Bingbot, crawl so the engine can retrieve and cite you in live answers. Blocking these is what removes you from AI results. OpenAI states plainly in its crawler documentation that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers.
Blocking GPTBot will not protect you from ChatGPT, but blocking OAI-SearchBot will give you protection. If you misunderstand that, you may either lose confidential data you were supposed to help keep confidential, or become invisible in the answers you wanted to receive. The optimizing for ChatGPT Search guide describes OpenAI bots in more detail, while optimizing for Claude, Grok, and Copilot covers the rest.
How do you set up robots.txt for AI?
Your configuration depends on your goal. Here are the three common ones.
To maximize AI visibility
Permit the response bots so that each engine has the chance to obtain and quote you. This is the best configuration for the majority of brands that seek to be recognized in AI.
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
Also confirm Googlebot and Bingbot are not disallowed anywhere, since they feed AI Overviews, AI Mode, and Copilot. Being crawlable is the entry ticket to being cited, and clean structured data helps engines parse you once they can.
To block training but stay citable
Many brands are searching for this kind of nuanced setup: keep your work outside of the training of models while being present in the AI answers. Stop the training bots and enable the answer bots.
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
Do not remove OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, and Bingbot. You lose nothing so far as visibility is concerned and you can turn off training. This is the winning situation, and it is achievable just because of independence of these settings.
To block everything
You can prevent all AI bots from accessing your content, but be aware of the implications. Doing this will cause you to be missing out on AI search results from multiple engines meaning you will suffer visibility penalty as more information is consumed through such channels.
Use this option on content that is really supposed not to be part of AI search results and be aware of the fact that anyway, professional scrapers will bypass this file.
Avoid the catch-all trap
The most prevalent accident is a broad rule that is designed to prohibit training while acting as a barrier to the answer bots. A blanket ban or a rule that inadvertently matches OAI-SearchBot can eliminate your AI visibility while assuming that you only opted out of training.
Make sure to use accurate names for bots for AI directives after you have changed your file. It is the problem of one wrongly chosen wording that makes it necessary to choose between getting no experience in training the system and getting erased from ChatGPT.
How do you enforce robots.txt, and what are its limits?
Robots.txt works on an honor system, so it is only able to prevent polite bots from crawling the site. To stop a bot from crawling the site when the website has a robots.txt file, you need to implement CDN or WAF-based solutions like Cloudflare that can enforce rules to block bots.
If you allowlist or block by bot, you should check if you are using the right crawler and not just accept the user-agent string, as bots can easily fake this string. The main providers share their IP ranges with you and you can always use reverse DNS to check if the bot is actually a bot.
Also, remember that the llms.txt file that you might have come across is an idea that serves as a guide for AI systems to show them your best content and is mentioned in our guide to llms.txt. It does not replace robots.txt, rather complements it.
Frequently Asked Questions
Does blocking GPTBot remove me from ChatGPT?
No. GPTBot is only responsible for determining whether your content is permitted to educate OpenAI's models. How you appear in ChatGPT's search results is determined by a separate bot, OAI-SearchBot, so the two settings differ. Blocking GPTBot keeps you visible while deactivating OAI-SearchBot is what removes you from ChatGPT.
How do I block AI bots from training on my content but stay visible in AI answers?
Prevent training bots from being active but enable answer bots. GPTBot, Google-Extended, ClaudeBot, and CCBot need to be disabled while permitting OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot to continue. Such independent settings will protect your content from being used in model training while allowing it to appear in a real-time AI answer in various platforms.
Will robots.txt actually stop AI bots?
Only those who accept it should follow. Robots.txt is a voluntary rule that major AI companies honor, so it applies only to trustworthy bots. It is not a security wall, and some scrapers ignore it. To stop non-compliant crawlers, use a CDN or WAF and verify bots by IP address or reverse DNS, not the user-agent string.
Which AI bots should I allow for visibility?
Make sure to let OAI-SearchBot to ChatGPT through, while allowing Claude-SearchBot for Claude, accept PerplexityBot for Perplexity, and do not block Googlebot and Bingbot. Otherwise they cannot feed AI Overviews, AI Mode, and Copilot. These crawling bots let each engine index and cite you, so blocking any of them costs you visibility.
Does blocking Google-Extended hurt my Google visibility?
No, not your Search or AI Overview visibility. Google-Extended controls whether your content is used for Gemini training and some grounding, separately from Googlebot, which handles Search indexing and your eligibility for AI Overviews and AI Mode. You can block Google-Extended to opt out of training while remaining fully visible in Google Search and its AI answers.
What is the difference between robots.txt and llms.txt?
Robots.txt tells crawlers which parts of your site they may access, and it is widely honored. Llms.txt is a separate, emerging proposal that points AI systems to your most important content. They serve different purposes, adoption of llms.txt is still limited, and it does not replace robots.txt. Treat llms.txt as an optional complement.
Where Unveilr fits
A robots.txt document must not be considered "set-and-forget" in the age of AI. New bots are developed and organizations are renaming their crawlers which means that what seems like an innocent change can seriously affect your visibility. Thus, it is paramount to conduct regular checks to monitor how the engines treat you.
The check is included in the Unveilr loop: tracking how AI engines respond and refer to you, noticing the negative effect of a crawler or a configuration change on your visibility, making changes in configuration settings or content to restore visibility, and checking back to confirm.
In a particular internal study of a D2C brand, it managed to go from the ninth most common source of citations to the first, with the brand's ChatGPT figure improving from 3.3 percent to 44.7 percent as a result.
To learn about the strategic aspects of this technical work, refer to the GEO playbook, the complete guide to answer engine optimization, the get cited by ChatGPT, and the measure your AI share of voice.