Quick answer: llms.txt is a proposed standard — a plain Markdown file at your site’s root (yoursite.com/llms.txt) that lists your most important pages so large language models can find and understand them. Think “robots.txt for AI,” but for guidance, not blocking. It’s trivial to create. The catch: real-world data shows AI search engines (ChatGPT, Perplexity, Google AI Overviews) barely fetch it today, and never look for it on their own. So it’s optional — auto-generate one if your platform offers it, but don’t spend real effort on it for AI visibility.
If you’ve spent any time in GEO (generative engine optimization) circles in 2026, you’ve heard “add an llms.txt file” repeated like gospel. But most explainers skip the two questions that actually matter: what is it, and does anything read it? This guide answers both — with our own test data, not just theory. (This is the hub for our llms.txt coverage; we link the deep-dives as we go.)
What is llms.txt, exactly?
llms.txt is a Markdown file you place at the root of your domain — for example, https://yoursite.com/llms.txt. It’s a curated, human-readable map of your site: your most important pages, short descriptions, and links, written so a large language model can quickly grasp what your site is about and where the good content lives.
The proposal (introduced by Jeremy Howard of Answer.AI in late 2024) addresses a real problem: LLMs have limited context windows and struggle to parse cluttered HTML, navigation, ads, and scripts. An llms.txt file is supposed to hand them a clean summary instead of making them crawl your whole site. The idea is sensible. Whether the tools that matter actually consume it is a separate question — and the one we test below.
What does an llms.txt file look like?
It’s just Markdown. A minimal example:
# GeoParrot
> GEO Lab — we run experiments to test what AI search engines actually cite, then publish the data.
## Core guides
- [What Is GEO?](/what-is-generative-engine-optimization-geo-2026-playbook/): The 2026 playbook for getting cited by AI search.
- [GEO vs SEO vs AEO](/geo-vs-seo-vs-aeo-difference/): How the three disciplines differ.
## Data & studies
- [Is llms.txt Worth It?](/is-llms-txt-worth-it-2026-data/): What 137,000 websites reveal.
## Optional
- [Contact](/contact/)The format is loose: an H1 with your site name, an optional blockquote summary, then H2 sections of links with one-line descriptions. Some sites also publish llms-full.txt, a longer version with full page contents inlined.
llms.txt vs robots.txt vs sitemap.xml — what’s the difference?
They’re often confused, but they do different jobs:
| File | Audience | Purpose | Read by AI search today? |
|---|---|---|---|
| llms.txt | LLMs / AI agents | Guide them to your key content (a curated map) | Rarely |
| robots.txt | All crawlers | Allow/block what bots may access | Yes (widely respected) |
| sitemap.xml | Search crawlers | List all URLs for discovery/indexing | Yes |
Key distinction: robots.txt and sitemap.xml are established standards that crawlers actively look for. llms.txt is a proposed one that, as of 2026, most AI systems don’t probe for unless told to.
Does llms.txt actually work for AI visibility?
This is where theory meets data — and the data is blunt. The largest study to date analyzed server logs from 137,210 domains (Ahrefs, May 2026):
- 28% of those domains publish an llms.txt file.
- 97% of those files received zero requests in a month.
- The AI search/retrieval bots you actually want (OAI-SearchBot, PerplexityBot, Claude’s search crawler) were just 1.1% of the few fetches that happened.
- No AI bot ever probed for a missing llms.txt file — agents fetch it only when explicitly pointed at it.
That last point is decisive: if AI search engines don’t even look for the file, publishing one can’t help you get cited. This also matches the platforms’ own statements — Google has confirmed it doesn’t use llms.txt. We break down the full study, and our own published-and-tested file, in Is llms.txt Worth It? What 137,000 Websites Reveal.
So who does read llms.txt?
It isn’t completely ignored — it’s just read by the wrong audience for GEO. Among the small slice of fetches that did happen, the biggest consumers were coding agents (Claude-Code, GPTBot) pulling documentation on request — not answer engines building citations. So llms.txt is genuinely useful in one case: your customers are developers using AI coding tools that you want pulling your docs. For everyone chasing ChatGPT/Perplexity citations, it’s noise.
Do you need an llms.txt file?
A simple decision rule:
- You ship developer docs / an API / a dev tool → Yes, publish one. Coding agents are your real audience.
- Your platform auto-generates one (a plugin, a framework) → Keep it. Zero effort, no harm.
- You’re a content site / SMB chasing AI search citations → Optional and low priority. It won’t move your AI visibility. Spend the hour on content structure, schema, and being a genuinely citable source instead.
What does move AI visibility is covered in our pillar: What Is Generative Engine Optimization (GEO)? — and if you’re weighing where to spend effort, GEO vs SEO vs AEO.
How do you create an llms.txt file?
It takes about ten minutes:
- Create a plain text file named
llms.txt. - Add an H1 with your site name and a one-line summary blockquote.
- List your most important pages under H2 sections, each link with a short description.
- Upload it to your site’s root so it resolves at
yoursite.com/llms.txt. - Verify it loads in a browser, then leave it — no ongoing maintenance needed.
On WordPress specifically, we walk through doing this (with our real file) in How to Add an llms.txt File to WordPress.
The GEO Lab verdict
llms.txt is a clean idea that solves a real problem — clutter — for the wrong audience if your goal is AI search citations. The hype outran the evidence. Our position, backed by the 137k-domain data and our own test: create one if it’s free, skip the manual effort, and put your energy into being the kind of source AI engines already cite. When the standard gets real adoption from search bots, we’ll re-test and update this page. That’s the GEO Lab promise — we don’t repeat claims, we check them.
Frequently asked questions
Is llms.txt an official standard?
No. It’s a community proposal introduced by Jeremy Howard in late 2024. Unlike robots.txt and sitemap.xml, no major AI platform has officially committed to reading it, and Google has said it doesn’t use it.
Where do I put the llms.txt file?
At your domain’s root, so it’s reachable at https://yoursite.com/llms.txt — the same place as robots.txt.
Will llms.txt help me get cited by ChatGPT or Perplexity?
Not measurably, based on current data. AI search bots account for ~1.1% of llms.txt fetches and never probe for the file on their own, so it can’t drive citations today.
What’s the difference between llms.txt and llms-full.txt?
llms.txt is a concise map of links and descriptions. llms-full.txt inlines the full text of those pages into one file, so an LLM can ingest everything without following links.
Sources
- Ahrefs — “We Analyzed llms.txt Usage Across 137,210 Domains” (Louise Linehan & Xibeijia Guan, May 2026).
- llmstxt.org — the original llms.txt proposal (Jeremy Howard, Answer.AI, 2024).
- Google Search Central — guidance on generative AI features and llms.txt (2026).
Leave a Reply