How is llms.txt different from robots.txt or sitemap.xml?

Different jobs entirely. robots.txt controls what crawlers may access, sitemap.xml lists what pages exist for discovery, and llms.txt offers a clean condensed summary so a model can understand your site without parsing full HTML. Only robots.txt is about permission.

Does llms.txt block AI crawlers?

No — it grants nothing and blocks nothing. If you want to restrict AI crawler access, that's robots.txt, and even then only for crawlers that choose to honour it.

Do I still need a sitemap if I have llms.txt?

Yes. They serve different audiences and purposes — sitemap.xml helps search crawlers discover and prioritise your URLs, while llms.txt helps models comprehend your content. Neither substitutes for the other.

Is llms.txt worth adding to my site?

If you can generate it from existing content, the cost is low enough to be worth it. If it means hand-maintaining a second description of your site that will drift out of date, the value is questionable — a stale llms.txt is worse than none.

Does llms.txt help SEO rankings?

Not directly — it isn't a ranking signal for traditional search. Its purpose is making your content easier for AI answer engines to read and cite accurately, which is a distinct goal from ranking.

Concepts

llms.txt vs robots.txt vs sitemap.xml: three files, three completely different jobs

TL;DR

Three plain files at your site root, three unrelated purposes: robots.txt controls access, sitemap.xml aids discovery, and llms.txt offers comprehension. Confusing them leads to real mistakes — most commonly expecting llms.txt to block AI crawlers, which is robots.txt's job and something llms.txt does not do at all.

The three in one table

File	Question it answers	Audience
`robots.txt`	What are you allowed to crawl?	Crawlers that choose to honour it
`sitemap.xml`	What pages exist and when did they change?	Search engine crawlers
`llms.txt`	What is this site about, in a form a model can read cheaply?	AI answer engines and agents

None of the three enforces anything. robots.txt is a convention well-behaved crawlers follow, not access control — if you need something private, authenticate it.

robots.txt — permission

The oldest of the three, and the only one about access. It tells crawlers which paths they may fetch, and increasingly carries directives aimed at specific AI crawlers by user-agent. If your intent is to keep AI systems out of your content, this is the file that expresses it — and even then only for crawlers that honour it.

sitemap.xml — discovery

A machine-readable index of your URLs with metadata about when each changed. It doesn't influence ranking and it doesn't grant permission; it helps crawlers find pages efficiently, which matters most on large sites or ones with weak internal linking. A stale `lastmod` on every URL is worse than none, because it trains crawlers to ignore the signal.

llms.txt — comprehension

The newest and most misunderstood. It's a Markdown file offering a clean, condensed version of what your site covers, so a model can understand it without crawling and parsing your full HTML — navigation, scripts, cookie banners and all.

Critically: it grants nothing and blocks nothing. It's an offer of clarity, not a permission layer. Adoption is not universal, so treat it as cheap insurance rather than a guaranteed channel.

Is llms.txt worth adding?

Cost is low if you generate it from content you already have rather than maintaining it by hand.
The failure mode is drift — a hand-written llms.txt describing a site you've since changed is worse than not having one.
It won't fix thin content. A clean summary of a site with nothing to say produces a clean summary of nothing.

We generate ours from the same content arrays that render the pages, so it can't drift out of sync — that's the pattern worth copying, more than the file itself. See what llms.txt is for the implementation.

Frequently asked questions

How is llms.txt different from robots.txt or sitemap.xml?: Different jobs entirely. robots.txt controls what crawlers may access, sitemap.xml lists what pages exist for discovery, and llms.txt offers a clean condensed summary so a model can understand your site without parsing full HTML. Only robots.txt is about permission.
Does llms.txt block AI crawlers?: No — it grants nothing and blocks nothing. If you want to restrict AI crawler access, that's robots.txt, and even then only for crawlers that choose to honour it.
Do I still need a sitemap if I have llms.txt?: Yes. They serve different audiences and purposes — sitemap.xml helps search crawlers discover and prioritise your URLs, while llms.txt helps models comprehend your content. Neither substitutes for the other.
Is llms.txt worth adding to my site?: If you can generate it from existing content, the cost is low enough to be worth it. If it means hand-maintaining a second description of your site that will drift out of date, the value is questionable — a stale llms.txt is worse than none.
Does llms.txt help SEO rankings?: Not directly — it isn't a ranking signal for traditional search. Its purpose is making your content easier for AI answer engines to read and cite accurately, which is a distinct goal from ranking.