Is llms.txt Worth It? What 137,000 Websites Reveal (2026 Data)

llms.txt has been one of the most hyped ideas in GEO: a simple Markdown file at your site’s root that “tells AI models what matters” on your site. The pitch is seductive. But does it actually move the needle on AI visibility? We now have real data — and we’ve published our own llms.txt to test it. Here’s the honest verdict.

What is llms.txt, in one line?

llms.txt is a proposed standard: a Markdown file (e.g., yoursite.com/llms.txt) that lists your most important pages and context, so large language models can find and understand your content more easily. Think of it as a “robots.txt for AI,” but for guidance rather than blocking. The idea is reasonable. The question is whether anything actually reads it.

Do AI search engines actually use llms.txt?

Short answer: not in any measurable way. Ahrefs (Louise Linehan and Xibeijia Guan) studied server logs from 137,210 domains in May 2026. The findings are blunt:

• 28% of those domains publish an llms.txt file.
• 97% of those files got zero requests in the month studied.
• Of the 3% that got any traffic, 96% of requests were bots, not humans.
• The AI retrieval/search bots — the ones that build answers in Perplexity and ChatGPT Search — were just 1.1% of fetches. Ahrefs notes OAI-SearchBot, PerplexityBot, and Claude’s search crawler combined made “only a couple of hundred fetches across thousands of sites.”
• No AI bot ever probed for a missing llms.txt file. Agents fetch it when explicitly directed — never speculatively.

That last point is the killer. If AI search engines don’t even look for the file, publishing one can’t help you get cited. This lines up with Google’s own position — Google has confirmed it doesn’t use llms.txt.

So who is fetching llms.txt?

Here’s the twist: the file isn’t completely ignored — it’s just read by the wrong audience for GEO. Among the small slice of fetches that happened, the biggest consumers were coding agents like Claude-Code and GPTBot, not answer engines.

• AI agents — 10.5% (Claude-Code was second only to GPTBot). These are coding/dev assistants pulling docs on request.
• AI training crawlers — 5.3%.
• AI assistants — 2.5%.
• AI search / retrieval bots — 1.1% ← the audience GEO actually cares about.

In other words, llms.txt today is mostly useful if your customers are developers using coding agents who you want pulling your API docs. For an SMB, a local business, or a marketer trying to get mentioned in an AI answer, it does essentially nothing.

Should you delete the llms.txt you already have?

No — leave it. It costs nothing to keep, it won’t hurt you, and the standard might gain traction later. As Ahrefs put it: “Either we’re watching the early scaffolding of a real standard, or we’re watching the SEO industry prove it can productize anything.” Our stance: keep a low-effort one, but don’t expect it to drive citations.

1. Don’t hand-maintain it. Let your CMS or a plugin generate it automatically. Here’s our WordPress walkthrough if you want a zero-effort one.
2. Link to it from your HTML so the agents that do use it (coding assistants) can discover it — they only fetch when directed.
3. Don’t spend an afternoon curating it. The ROI isn’t there for AI search visibility.

What actually gets you cited by AI (do this instead)

The hour you’d spend perfecting llms.txt is far better spent on the things AI engines genuinely use to choose sources:

• Answer-first structure. Lead with a clear, quotable answer. See how to structure content so AI engines quote it.
• Schema markup. Article, FAQ, and HowTo schema help machines parse your meaning.
• Be a corroborated, trusted source. Citations follow reputation and consistent mentions across the web, not a config file.
• Measure what’s working. Track your real AI citation share — start with our 10-point GEO audit and the best GEO tools for 2026.

Frequently asked questions

Is llms.txt worth it in 2026?

For AI search visibility, no — the data shows AI answer engines barely fetch it (1.1% of requests) and never look for it on their own. It’s only worth it if your audience uses coding agents that pull your docs. Keep an auto-generated one; don’t invest manual effort.

Does ChatGPT or Perplexity read llms.txt?

Almost never. In a 137,210-domain study, search crawlers like OAI-SearchBot and PerplexityBot made only a couple hundred fetches across thousands of sites — and never probed for the file when it was missing.

Will llms.txt help my SEO or rankings?

No. It’s not a ranking signal for Google search or for AI Overviews. Content quality, structure, and schema are what move AI visibility.

The bottom line

llms.txt is a sensible idea that the AI ecosystem hasn’t adopted — yet. The 137,000-site data is clear: it doesn’t drive citations today. Keep a hands-off version in case the standard matures, then put your energy where the evidence points — answer-first content, schema, trust, and measurement. That’s what gets you into AI answers.

Sources

Data from Ahrefs, “We Studied 137,210 Domains. Here’s the Truth About llms.txt” by Louise Linehan and Xibeijia Guan (server-log analysis, May 2026). Percentages quoted as reported. Our own llms.txt observations are first-hand.

Related

• Google Confirms It Doesn’t Use llms.txt
• How to Add an llms.txt File to WordPress (if you still want one)
• How to Structure Content So AI Engines Quote It
• Best GEO Tools 2026