The llms.txt specification is open for community input, so we treat the file as optional (1). The proposal's author reports that thousands of sites publish an llms.txt file. Recala does not guarantee rankings or AI citations from it.
Should my website have an llms.txt file in 2026?
AnswerDotAI's proposal adds a markdown /llms.txt file to websites to provide LLM-friendly content (1). Its author reports that thousands of sites publish one, that documentation platforms generate it automatically, and that Chrome's Lighthouse audits sites for one. OpenAI, Anthropic and Gemini each publish an llms.txt for their own developer docs (1).
The author says llms.txt information is used on demand, when an agent needs information about a topic while assisting a user (1). Its author expects llms.txt to be used mainly for inference rather than training, and reports that this is how it has been used. The specification remains open for community input, and the admitted sources do not show that Google or ChatGPT search uses llms.txt to choose citations.
How is llms.txt different from robots.txt and sitemap.xml, and which file controls whether OpenAI and Google crawlers reach my site?
robots.txt and llms.txt have different purposes: robots.txt lets automated tools know what access to a site is considered acceptable, while llms.txt information is used on demand when an agent needs information (1). Before crawling a site, Google's crawlers download and parse its robots.txt file to extract which parts of the site may be crawled (2). OpenAI documents OAI-SearchBot and GPTBot as independent robots.txt settings, and disallowing GPTBot indicates that crawled content should not be used for training OpenAI's generative AI foundation models (3). Google's crawlers only apply the Robots Exclusion Protocol to their automated crawling, so the protocol is not applicable to crawlers controlled by users (2). OpenAI notes that ChatGPT-User is not used to determine whether content may appear in Search, and that OAI-SearchBot should be used for managing Search opt-outs (3).
llms.txt does not grant or block crawler access; robots.txt rules for bots such as OAI-SearchBot and GPTBot do that (1). sitemap.xml lists indexable pages, often too many to fit in an LLM context window, so it does not replace llms.txt. Keep your access decisions in robots.txt, and treat llms.txt as a separate overview file.
What is the minimal llms.txt I could publish?
The file is named llms.txt and sits at the root, /llms.txt, or at any subpath such as /docs/llms.txt (1). A file covers the URLs under its path, and where more than one applies, agents should use the most specific one. The only required section is an H1 with the name of the site or project, so the smallest valid file is a single heading.
A fuller file follows a fixed order: an optional byte-order mark, the H1, a blockquote with a short summary, then zero or more markdown sections (1). Its links should point to LLM-friendly content such as markdown versions of your pages. The file itself stays small enough to fit in context, and the detail sits behind the links.
Minimal llms.txt: what the proposal specifies (1)
Minimal llms.txt: what the proposal specifies
| Element | What the proposal specifies |
|---|---|
| File name and location | Named llms.txt, at the root /llms.txt or any subpath; it covers URLs under its path and the most specific file applies |
| H1 with site or project name | The only required section |
| Blockquote summary | A short summary with the key information needed to understand the rest of the file |
| Markdown sections with links | Zero or more sections; links should point to LLM-friendly content such as markdown versions of pages, and the file stays small enough to fit in context |
Next step
Continue with Recala to turn this plan into a concrete next step.
Sources
The /llms.txt file, v2, AnswerDotAI, 3 September 2024github.com
How Google interprets the robots.txt specification, Google for Developers, last updated 23 October 2026developers.google.com
Overview of OpenAI Crawlers, OpenAI, accessed 5 November 2026developers.openai.com
recala.co