llms.txt is a proposed standard file, placed at yourdomain.com/llms.txt, that gives AI models a clean, markdown-formatted map of your most important pages. Jeremy Howard of Answer.AI proposed it in September 2024. The goal is to help LLMs find and read your best content without wading through HTML, ads, and navigation clutter.
That is the pitch. Whether it does anything useful yet is a separate question, and the honest answer in mid-2026 is: probably not much. This article covers what the file is, how it differs from robots.txt, and whether you should bother adding one.
What llms.txt is meant to do
The problem llms.txt tries to solve is real. When an AI model wants to use a webpage, it has to convert messy HTML into plain text.
Pages are full of menus, cookie banners, scripts, and boilerplate that bury the actual content. Model context windows are also limited, so feeding an entire site into a prompt is wasteful.
llms.txt proposes a fix: a single markdown file that lists your key pages with short descriptions, curated by you. Instead of guessing which URLs matter, an AI agent could read one file and jump straight to clean content. Think of it as a hand-picked table of contents written for machines rather than people.
There is also a companion file, llms-full.txt, which contains the full text of your important pages in one markdown document. The idea is that a model could ingest the whole thing in a single pass.
The format
The spec is deliberately simple. An llms.txt file is plain markdown with a specific structure:
- An H1 with the site or project name (the only required element)
- A blockquote with a short summary of what the site is
- Optional paragraphs giving context
- One or more H2 sections containing lists of links, each formatted as
[name](url): optional note
A section titled ## Optional has special meaning. Consumers can skip those links when they need a smaller context window. Here is a short example:
# Acme Analytics
> Acme Analytics is a self-serve product analytics tool for SaaS teams.
> These docs cover setup, the API, and common integration patterns.
## Docs
- [Getting Started](https://acme.com/docs/start): Install and send your first event
- [API Reference](https://acme.com/docs/api): Full REST API documentation
- [Integrations](https://acme.com/docs/integrations): Connect Segment, Stripe, and webhooks
## Optional
- [Changelog](https://acme.com/changelog): Release history
- [Brand Assets](https://acme.com/brand): Logos and color palette
That is the whole format. You can write one by hand in a few minutes, or generate it from your sitemap with one of the many free tools that now exist.
llms.txt vs robots.txt
People keep comparing the two because both are plain text files that sit at the root of your domain. They do almost opposite jobs.
robots.txt is a gatekeeper that tells crawlers where they may not go. llms.txt is a curator that recommends where AI models should look. One restricts, the other invites.
| robots.txt | llms.txt | |
|---|---|---|
| Purpose | Control crawler access | Curate content for LLMs |
| Audience | Search and AI crawlers | LLMs / AI agents |
| Action | Blocks or allows paths | Lists and describes key pages |
| Format | Directive syntax (User-agent, Disallow) | Markdown |
| Location | /robots.txt |
/llms.txt |
| Adoption | Universal, decades old | Proposed 2024, not officially supported |
| Honored by major bots? | Yes (by convention) | No confirmation from any major provider |
The key difference: robots.txt is a long-established convention that crawlers actually respect. llms.txt is a proposal that, so far, the big AI companies have not agreed to follow.
Does it actually work today?
Here is the part most vendor blogs skip. As of mid-2026, no major AI provider has confirmed that it reads llms.txt for general web answers. Not OpenAI, not Google, not Anthropic.
What Google says
Google has been the most direct. Gary Illyes said at Search Central Live in July 2025 that Google does not support llms.txt and has no plans to.
John Mueller compared it to the old keywords meta tag, a self-declared signal that search engines abandoned because it was trivially gamed. His logic was blunt: if a site already serves the real content, why would a bot trust a separate file it has to cross-check for spam anyway?
Google's own guidance for AI features now lists llms.txt by name as a tactic that does not help. By Google's account it neither helps nor harms your ranking. The signals that actually move Google's AI surfaces are covered in our Google AI Overviews guide.
OpenAI, Anthropic, and the crawler evidence
OpenAI has not announced support either. Server-log studies through 2025 found that GPTBot and OpenAI's crawlers do not request /llms.txt during normal site visits. That is the clearest possible evidence that they are not using it.
The broader picture is stark. One study found that about 97 percent of llms.txt files are never fetched by AI crawlers at all.
Anthropic is the interesting case. The company publishes both llms.txt and llms-full.txt at its documentation site, which suggests at least internal interest. But publishing a file is not the same as confirming that Claude consumes it during inference, and Anthropic has not made that claim publicly.
Who actually uses it today
So who is adopting it? Mostly developer-tooling and documentation platforms. Companies like Mintlify, Vercel, and Cloudflare serve llms.txt files, often so that coding assistants can pull their docs more cleanly.
Thousands of sites now have one. That growth is real, but it reflects supply, not demand.
Lots of people are publishing the file. Almost no one with a major model has confirmed they read it.
The honest case, for and against
The case in favor is modest. The file is cheap to add, it does no harm, and if adoption tips toward official support later, you are already set up.
The case against is that it is unverified, easy to abuse via cloaking (showing AI one thing and users another), and it duplicates information crawlers can already get from your pages and structured data. Both of those are fair points.
How to create one
If you want to add an llms.txt file, it takes about ten minutes.
- List your most important URLs. Pull the 10 to 30 pages you most want an AI model to read: core docs, product pages, your best guides. Skip thin or duplicate content.
- Write the file in markdown. Start with an H1 (your site name), a one or two line blockquote summary, then group links under H2 sections like
## Docsor## Guides. Add a short note after each link. - Move less critical links under
## Optional. This lets a model skip them when context is tight. - Save it as
llms.txtand upload it to your site root so it resolves atyourdomain.com/llms.txt. On most stacks that means dropping it in your public or static folder. - (Optional) Add
llms-full.txtwith the full markdown text of your key pages if you want to offer a complete-context version. - Keep it current. A stale file pointing at dead URLs is worse than no file. Update it when your important pages change.
Use the example above as a template. There are also generators that build a draft from your sitemap, which is a reasonable starting point if you have a large site.
Who should bother (and who shouldn't yet)
This comes down to your situation rather than a blanket yes or no.
Worth doing now if you run documentation, a developer tool, or an API product. Coding assistants and doc-aware agents are the one place llms.txt sees real use today, and a clean file genuinely helps them pull your content. The audience exists.
Low priority if you run a typical marketing site, blog, or ecommerce store. There is no confirmed payoff for general AI search visibility, and your effort is better spent on things that demonstrably move AI citations: clear answer-first content, solid structure, and accurate schema markup.
If you want a real edge in AI answers, content structure and schema markup are where the evidence actually points.
The reasonable middle path: if it costs you ten minutes and you like being early, add one. Just don't expect it to lift your citations on its own, and don't let it distract from the fundamentals of answer engine optimization.
The bottom line
llms.txt is a sensible idea waiting for the major AI providers to opt in. The format is clean, the file is cheap to add, and it has real traction in developer documentation.
But no major model has confirmed it reads the file for general answers, and Google has said outright that it will not. On its own it is not a working lever for AI visibility today.
Add one if you run docs or want to be early. For everyone else, your AI visibility depends on the content and structure of your actual pages, not a side file.
Sources
- Answer.AI: The /llms.txt proposal (Jeremy Howard, Sept 2024)
- llmstxt.org: The /llms.txt file specification
- Search Engine Land: Meet llms.txt, a proposed standard for AI website content crawling
- Search Engine Journal: Google Says llms.txt Comparable To Keywords Meta Tag
- Search Engine Land: No, llms.txt is not the 'new meta keywords'
Frequently Asked Questions
Does Google use llms.txt?
No. Google has stated publicly, through Gary Illyes and John Mueller in 2025, that it does not support llms.txt and has no plans to. Its guidance for AI features explicitly lists the file as a tactic that does not help. By Google's own account, adding one neither improves nor harms your ranking.
Does ChatGPT or Claude read llms.txt?
There is no confirmation from OpenAI that ChatGPT reads it, and server-log analysis suggests its crawlers do not request the file during normal visits. Anthropic publishes its own llms.txt but has not confirmed that Claude consumes it during inference. Treat support as unverified for both, and do not count on it.
Is llms.txt the same as robots.txt?
No. robots.txt controls which paths crawlers may access, and it is a decades-old convention that search and AI bots actually respect. llms.txt instead curates and describes your best content for AI models, and no major provider has confirmed it uses the file. One blocks access, the other only recommends reading. They serve different jobs and do not replace each other.
Will adding llms.txt hurt my SEO?
No. The file is ignored by search ranking systems, so it will not penalize you or change how your pages rank. The only real downsides are the time spent maintaining it and the risk of a stale file that points at outdated or dead URLs. Keep it current or skip it.
What is the difference between llms.txt and llms-full.txt?
llms.txt is a short index of links to your key pages, each with a brief description. llms-full.txt instead contains the full markdown text of those pages in one document, so a model can ingest everything in a single pass without following links. Neither file is confirmed to be read by major AI engines.
Should I add llms.txt to my site in 2026?
If you publish technical documentation or developer tools, yes, because doc-aware AI agents and coding assistants use it today. For most marketing sites, blogs, and stores, it is optional and low priority. It does no harm, but it is not a proven lever for AI search visibility, so treat it as a nice-to-have.
Where Unveilr fits
The durable wins in AI search do not come from betting on any single unproven file. They come from measuring what the engines actually cite, then adapting every time a new model ships. GPT, Gemini, Claude, Perplexity, and Grok each update on their own schedule, and citation patterns shift with them.
Unveilr is built for exactly that loop. Its agents scan the AI engines to see who gets cited, detect what is winning those citations, feed the findings back into your content, and re-scan on every new model launch so the gains compound.
A static side file cannot do that. A continuous feedback loop can.
The results are concrete. Working this way, a D2C brand moved from the ninth most-cited domain to the single most-cited source in AI answers, with its ChatGPT visibility rising from 3.3 percent to 44.7 percent. That is what tracking and adapting to real citation data can do, with or without an llms.txt file.