Quick Answer: llms.txt is a Markdown file at yourdomain.com/llms.txt that gives AI models a short, curated map of your most important pages. Proposed in September 2024, it is cheap to add and useful for documentation sites, but no major AI assistant has confirmed it reads the file for general answers.
llms.txt is a proposed standard file, placed at yourdomain.com/llms.txt, that gives AI models a clean, Markdown-formatted map of your most important pages. Jeremy Howard of Answer.AI proposed it in September 2024, and the llmstxt.org specification is now on version 2.
This page covers what the file is, the exact format, ten real files described from what their servers returned, how to generate one, how to validate it, and whether you should bother at all.
What is llms.txt?
llms.txt is a hand-picked table of contents written for machines. It lists the pages you most want an AI model to read, each with a one-line note, so an agent can skip menus, scripts and boilerplate.
The problem it targets is real. A model reading a web page has to strip messy HTML, and context windows are limited, so feeding a whole site into a prompt is wasteful. One curated file lets an agent jump straight to clean content.
A companion file, llms-full.txt, carries the full text of those pages in one document. The idea is that a tool can ingest everything in a single download rather than following links.
What does an llms.txt file contain?
A valid file contains, in this order: an H1 with the site name, a blockquote summary, optional prose, then H2 sections of Markdown link lists. Only the H1 is required.
| Order | Section | Required | What goes in it |
|---|---|---|---|
| 1 | H1 | Yes | The project or site name, nothing else |
| 2 | Blockquote | No, but expected | One short summary carrying the facts needed to read the rest |
| 3 | Free Markdown | No | Paragraphs or lists with context; no headings here |
| 4 | H2 file lists | No | Each item is [name](url), optionally followed by : notes |
An H2 named Optional has a reserved meaning: links an agent can skip when context is tight. Every other H2 name is yours to choose.
Where the file can live
The v2 spec allows the file at the site root or at any sub-path, and a file describes only the URLs beneath its own path. So /docs/llms.txt covers /docs/ only, and where two files apply, an agent should use the more specific one.
Markdown twins and discovery headers
The spec also asks for a Markdown twin of each important page, either page.html.md or page.md, and the links in llms.txt should point at those twins. To help agents find both, v2 recommends a Link header or <link> element with rel="alternate" type="text/markdown" for the twin.
Real llms.txt examples
The ten files below were fetched on 16 September 2026, and each description reflects what the server returned. "Spec shape" means H1, blockquote and H2 link lists in that order.
| File | Lines | H2s | Spec shape | What it actually contains |
|---|---|---|---|---|
| developers.openai.com/llms.txt | 51 | 4 | Yes | Blockquote, then links to per-product child llms.txt files |
| docs.anthropic.com/llms.txt | 698 | 3 | Partly | One-line prose summary instead of a blockquote, then 600-plus page links |
| ai.google.dev/gemini-api/docs/llms.txt | 208 | 1 | Partly | H1, a plain tagline and one Docs section of about 200 links |
| www.fastht.ml/docs/llms.txt | 45 | 4 | Yes | The spec author's own file: blockquote, notes, Docs, API, Examples, Optional |
| docs.perplexity.ai/llms.txt | 216 | 2 | Yes | Blockquote, a long link list, then OpenAPI Specs and Optional |
| docs.stripe.com/llms.txt | 698 | 26 | Partly | Two prose paragraphs of agent instructions, then 26 product sections |
| developers.cloudflare.com/llms.txt | 137 | 9 | Yes | A blockquote explaining that each product links to its own llms.txt |
| vercel.com/docs/llms.txt | 64 | 9 | Yes | Blockquote, then When to use, Agent setup, Documentation and Optional |
| svelte.dev/llms.txt | 23 | 3 | Yes | Blockquote, then small, medium and full documentation bundles |
| nextjs.org/llms.txt | 125 | 5 | Yes | Blockquote and a "When to use nextjs.org" section, then Documentation |
What the good files have in common
The files that follow the spec closely are short. OpenAI's, Vercel's and Svelte's stay under 65 lines by pointing at child files or full-text bundles instead of listing every page.
Two patterns are worth copying. Next.js and Vercel open with a "when to use this site" section that tells an agent what the site is authoritative for, and Cloudflare's blockquote explains its own structure so an agent knows to follow one more hop.
Files that drift from the spec
Anthropic's and Stripe's files are useful but heavy, at 698 lines each, and both replace the blockquote with prose. Google's Gemini API file has only a tagline and one long list.
Each still parses, because the H1 is the only hard requirement. None gives an agent a two-line orientation before the links begin, which is the part worth getting right.
How to generate an llms.txt file
You can generate an llms.txt file three ways: let your platform build it, script it from your sitemap, or write it by hand from a template. Every route ends with you reading the output.
Option 1: use a built-in generator
Several platforms now generate the file automatically. Each row below was checked against the vendor's own documentation on 5 October 2026.
| Platform | What it generates | How to switch it on |
|---|---|---|
| Mintlify (docs) | llms.txt and llms-full.txt for every hosted site; splits indexes over 100,000 characters | Automatic |
| GitBook (docs) | llms.txt for published docs sites, per the spec's integrations list | Automatic |
| Yoast SEO (WordPress, Shopify) | llms.txt refreshed weekly from recent and cornerstone content, with manual page selection | Toggle in site feature settings; included in the free plugin |
| Wix | llms.txt with a business summary and key links | SEO & GEO in the dashboard, then Go to llms.txt |
Two cautions apply to every generator. Yoast's feature page says it picks recently updated content by default, which may not be your most important content, and Wix stops auto-updating the file once you edit it by hand.
Option 2: script a draft from your sitemap
If your platform has no generator, a short script turns your sitemap into a first draft. The one below pulls page titles from a sitemap and writes the spec shape; you then cut it down to the pages that matter.
import re, urllib.request
SITE, NAME = "https://example.com", "Example Co"
SUMMARY = "One sentence: what you do, for whom, where."
xml = urllib.request.urlopen(SITE + "/sitemap.xml").read().decode()
urls = re.findall(r"<loc>(.*?)</loc>", xml)[:30]
lines = [f"# {NAME}", "", f"> {SUMMARY}", "", "## Key pages", ""]
for u in urls:
html = urllib.request.urlopen(u).read().decode(errors="ignore")
t = re.search(r"<title>(.*?)</title>", html, re.S)
lines.append(f"- [{(t.group(1) if t else u).strip()}]({u})")
open("llms.txt", "w").write("\n".join(lines) + "\n")
A crawl-based draft lists what it can see, not what an agent should prioritise. Delete thin pages, write a real note after each link, and move secondary links under ## Optional.
Option 3: write it by hand from a template
The template below follows the v2 section order and fits a company site rather than a docs site. Replace the bracketed text and delete any section you cannot fill honestly.
# [Company name]
> [One sentence: what the company does, for whom, and where it operates.]
Key facts an agent needs before reading further:
- Founded [year], headquartered in [city].
- Products: [two or three named products with a five-word description each]
- Pricing is on [URL]; support is at [email or URL].
## Products
- [Product A](https://example.com/product-a.md): [what it is, who it suits]
- [Product B](https://example.com/product-b.md): [what it is, who it suits]
## Docs
- [Getting started](https://example.com/docs/start.md): [scope in ten words]
## Company
- [About](https://example.com/about.md): [founders, history, locations]
## Optional
- [Blog index](https://example.com/blog/llms.txt): [only if you want agents reading it]
Keep the file under 100 lines, put the pages you most want quoted in the first section, save it as llms.txt in your public or static folder, and update it when key pages change. A stale file pointing at dead URLs is worse than none.
How do you validate an llms.txt file?
Validate it in three layers: parse it, fetch it, then test it with an agent. There is no official hosted validator.
The spec's authors publish a Python package, pip install llms-txt, whose parse_llms_file function returns the title, summary and sections it can read. An empty summary or no sections means the file has the wrong shape.
Chrome's Lighthouse includes an llms.txt check in its agentic browsing audits. Per the audit documentation, it flags a page only when the server errors on the request, and a 404 is marked Not Applicable, so passing proves the file is served, not that it is good.
- Fetch the file with curl and confirm a 200 status with a
text/plainortext/markdowncontent type, nevertext/html. - Confirm the first non-blank line is a single H1 and the next block is a blockquote.
- Fetch every link and confirm each resolves.
- Give an agent only the file and ask it your top five product questions.
llms.txt vs robots.txt
robots.txt tells crawlers where they may not go; llms.txt recommends where AI models should look. One restricts, the other invites.
| robots.txt | llms.txt | |
|---|---|---|
| Purpose | Control crawler access | Curate content for LLMs |
| Format | Directive syntax (User-agent, Disallow) | Markdown |
| Adoption | Universal, decades old | Proposed 2024, not officially supported |
| Honoured by major bots | Yes, by convention | No confirmation from any major provider |
Crawler access still lives in robots.txt, so check your robots.txt rules for AI bots before worrying about llms.txt. Blocking a search crawler there removes you from that engine's answers whatever your llms.txt says.
Does llms.txt actually work today?
Not for general AI search, as far as anyone has confirmed. No major AI provider has said its consumer assistant reads your llms.txt when answering questions.
What Google says
Google says its Search does not use the file. Its guide to generative AI features, last updated 10 July 2026 and read on 5 October 2026, lists "LLMS.txt files and other 'special' markup" among things you can ignore, which covers AI Overviews and AI Mode.
That matches what Google staff said in 2025: John Mueller compared it to the keywords meta tag, a self-declared signal engines abandoned. By Google's account the file neither helps nor harms rankings; the signals that do move its answers are covered in how to rank in Google AI Overviews.
OpenAI, Anthropic and Perplexity
OpenAI, Anthropic and Perplexity all publish llms.txt files for their own developer docs, as the table above shows. None has published a statement that its assistant reads yours during inference.
Real use today is concentrated in coding assistants and doc-aware agents. Thousands of sites publish a file; that growth reflects supply, not confirmed demand.
Who should add an llms.txt file?
Add one now if you run documentation, a developer tool or an API product, because coding agents genuinely use it. For a typical marketing site, blog or store, it is optional and low priority.
The file is cheap, does no harm and leaves you ready if support arrives. The case against is that it is unverified, open to cloaking, and duplicates what crawlers already get from your pages.
If you want a real edge in AI answers, spend the effort where evidence points: content structured for AI extraction, accurate schema markup and the fundamentals of generative engine optimization. Then measure what the engines actually cite rather than auditing which files you published.
