robots.txt for AI Bots in 2026

One single line of a text file may efficiently erase your brand from ChatGPT's responses. Such is the risk you take when setting robots.txt for AI crawlers, and many websites have done it without knowing it.

These days, AI crawlers are constantly scouring your site, and robots.txt is your main method of controlling these crawlers. Blocking all of them blindly is the lethal error that can cost you your online visibility, as some of those bots are the ones that allow you to be featured in AI search engine responses.

This guide will tell you which bots matter, along with one vital distinction that changes everything and how to configure your text file properly.

What does robots.txt control, and what does it not?

Robots.txt denotes a text file located at the top of the domain and gives information to different crawlers about the accessibility of the site's content. All well-known AI companies take robots.txt into account, making it a well-known way to grant bots some access to your site.

However, one should not forget about its limitations. It is an optional guideline and not a security mechanism. A directive only functions in case the bot decides to follow it as some robots completely disregard it.

It governs crawling, not what a model already learned, and it cannot claw back content that was crawled last year. For anything you truly need to stop, robots.txt is the first layer, not the only one.

Which AI bots are worth knowing?

The twelve user agents below cover the major AI engines, and each robots.txt token controls exactly one of them. The master table below lists every one, checked against the vendors' own crawler pages on 5 October 2026.

Master AI user-agent table

Two rows behave differently from the rest. Google-Extended is a control token with no user agent of its own, per Google's crawler documentation, and the user-triggered fetchers may not follow robots.txt: OpenAI says rules "may not apply" to ChatGPT-User, and Perplexity says Perplexity-User "generally ignores" them.

There are also a number of bots, like CCBot which comes from Common Crawl, Meta's crawlers, and the Amazon, Apple and ByteDance's crawlers, engaged in crawling for the main task of collecting training data. You may block these crawlers, but it will not in any way affect your AI answers searchability. Bytespider is known for its aggressive crawling practices and for violating rules.

Training bots vs answer bots: what is the difference?

This one key principle distinguishes the clever robots.txt from the one which conceals your identity. Here are the two classifications of AI that lead to opposite implications for your visibility.

Bots, such as GPTBot, Google-Extended, and ClaudeBot, search for material that is useful for training purposes. By preventing these bots from accessing your material, you can keep your information out of future training. However, it does not prevent you from being included in AI responses right now.

Answer bots, like OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, and Bingbot, crawl so the engine can retrieve and cite you in live answers. Blocking these is what removes you from AI results. OpenAI states plainly in its crawler documentation that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers.

Blocking GPTBot does not remove you from ChatGPT; blocking OAI-SearchBot does. Misread that and you either expose content you meant to keep out of training or vanish from the answers you wanted to appear in. OpenAI adds that a robots.txt change takes about 24 hours to reach ChatGPT search.

How do you set up robots.txt for AI?

Your configuration depends on your goal. Here are the three common ones.

To maximize AI visibility

Permit the response bots so that each engine has the chance to obtain and quote you. This is the best configuration for the majority of brands that seek to be recognized in AI.

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

Also confirm Googlebot and Bingbot are not disallowed anywhere, since they feed AI Overviews, AI Mode, and Copilot. Being crawlable is the entry ticket to being cited, and clean structured data helps engines parse you once they can.

To block training but stay citable

Many brands are searching for this kind of nuanced setup: keep your work outside of the training of models while being present in the AI answers. Stop the training bots and enable the answer bots.

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

Do not remove OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, and Bingbot. You lose nothing so far as visibility is concerned and you can turn off training. This is the winning situation, and it is achievable just because of independence of these settings.

To block everything

You can prevent all AI bots from accessing your content, but be aware of the implications. Doing this will cause you to be missing out on AI search results from multiple engines meaning you will suffer visibility penalty as more information is consumed through such channels.

Use this option on content that is really supposed not to be part of AI search results and be aware of the fact that anyway, professional scrapers will bypass this file.

Avoid the catch-all trap

The most prevalent accident is a broad rule that is designed to prohibit training while acting as a barrier to the answer bots. A blanket ban or a rule that inadvertently matches OAI-SearchBot can eliminate your AI visibility while assuming that you only opted out of training.

Make sure to use accurate names for bots for AI directives after you have changed your file. It is the problem of one wrongly chosen wording that makes it necessary to choose between getting no experience in training the system and getting erased from ChatGPT.

How do you enforce robots.txt, and what are its limits?

Robots.txt works on an honor system, so it is only able to prevent polite bots from crawling the site. To stop a bot from crawling the site when the website has a robots.txt file, you need to implement CDN or WAF-based solutions like Cloudflare that can enforce rules to block bots.

If you allowlist or block by bot, you should check if you are using the right crawler and not just accept the user-agent string, as bots can easily fake this string. The main providers share their IP ranges with you and you can always use reverse DNS to check if the bot is actually a bot.

Also, remember that the llms.txt file that you might have come across is an idea that serves as a guide for AI systems to show them your best content and is mentioned in our guide to llms.txt. It does not replace robots.txt, rather complements it.

What should you check after changing robots.txt?

Treat the edit like a deploy rather than a config tweak. Three checks catch almost everything that goes wrong.

Fetch the live file afterwards and read it line by line. A rule that looked correct in a diff can still match a bot you meant to allow, and the catch-all trap above is the usual culprit.

Then search the file for the five answer bots by name and confirm no broader rule sits above them. Exact spelling matters here, because a near-miss on OAI-SearchBot fails silently and nothing in your analytics will tell you.

Finally, watch the outcome rather than the file. Crawler access sits upstream of everything else, so when your share of voice shifts without a content change, robots.txt is the first place to look. Every tactic in the GEO playbook assumes this layer is already open.