Glossary
llms.txt vs robots.txt: what's the difference?
robots.txt tells crawlers which paths they may crawl, using allow and disallow rules, and is honoured by major search engines and AI crawlers. llms.txt is a proposed Markdown file that summarises a site and lists its key links for language models; it controls nothing. Use robots.txt for access and llms.txt as an optional guide.
Side by side
| robots.txt | llms.txt | |
|---|---|---|
| Purpose | Allow or block crawling | Curated summary and key links |
| Format | User-agent and Disallow lines | Markdown |
| Status | IETF standard (RFC 9309) | Proposal (2024) |
| Read by | Search engines, AI crawlers | Some agents and tools when pointed to it |
| Can block bots | Yes (for compliant bots) | No |
Using both
Set AI crawler access in robots.txt: allow AI search crawlers if you want to be cited. Publish llms.txt as a short, accurate guide to your most useful pages for agents and tools. Our site has both.
Common mistake
Putting "Disallow" rules in llms.txt, or expecting llms.txt to keep content out of AI training. It doesn't.
Example
A founder blocks GPTBot in robots.txt (training) while allowing OAI-SearchBot and PerplexityBot (search), and publishes llms.txt linking her docs, pricing and FAQ for agents.
FAQ
Do I need both?
robots.txt is essential; llms.txt is optional.
Does Google read llms.txt?
Google has said it doesn't use it.
Where do they go?
Both at your domain root.
Can I list my sitemap in llms.txt?
You can link it, but llms.txt is meant as a curated guide, not a full URL list.