llms.txt and robots.txt
Job
- robots.txt:
- Tells crawlers what they may fetch
- llms.txt:
- Points AI tools to the pages worth reading
Status
- robots.txt:
- A long-standing rule crawlers follow (Google)
- llms.txt:
- A proposal; support differs by tool
Blocks anything?
- robots.txt:
- Yes, for crawlers that obey it
- llms.txt:
- No
| robots.txt | llms.txt | |
|---|---|---|
| Job | Tells crawlers what they may fetch | Points AI tools to the pages worth reading |
| Status | A long-standing rule crawlers follow (Google) | A proposal; support differs by tool |
| Blocks anything? | Yes, for crawlers that obey it | No |
Google says no special file is needed to appear in its AI features (Google Search Central). So an llms.txt file is a small, cheap extra, not a fix on its own.
Should you add one?
If it takes an hour, yes: list your main pages with one honest line each. Do not expect it to change answers by itself. First make sure the crawlers can reach your pages, which our free AI crawler check shows.
Questions people ask
01Does llms.txt really work?
It is read by some tools and ignored by others, and no major AI company has said it uses the file to choose sources. Treat it as a tidy map, not a ranking lever.
02Should I add llms.txt to my website?
It does no harm and takes little time. Do the things that matter more first: crawler access, clear answers on your pages and structured data.