# llms.txt: Do You Need It and How to Build It Right
> What llms.txt is, who actually reads it (Ahrefs data, Google's position), how to generate it and serve .md pages, and what it does and does not do for GEO.
> Author: Roman Belov · Published: 2026-10-28 · Source: https://futurecraft.pro/blog/llms-txt-guide/
llms.txt is a Markdown file in the site root with a short description of the project and a list of key pages for language models and agents. Jeremy Howard proposed the format in September 2024. Since then, Yoast SEO and AIOSEO have added llms.txt generators, Anthropic, Perplexity, and Yandex publish the file for their docs, and Chrome Lighthouse has a dedicated audit. Yet no major AI platform has confirmed that it reads the file, and Google says outright that Search ignores it.
According to Ahrefs, 97% of llms.txt files got zero requests in May 2026. The other 3% were fetched mostly by SEO auditors and other bots: named AI bots accounted for 19.5% of requests.
This article covers what is confirmed about llms.txt and what is not, who actually needs the file, how to generate it from content and check it in CI, why .md versions of pages belong next to it, and how to measure the effect. The examples come from the futurecraft.pro blog: the generator, an incident with broken links, and cutting the file from 111 KB to 31.5 KB.
## What llms.txt is and how it differs from robots.txt and sitemap.xml
The specification lives at [llmstxt.org](https://llmstxt.org/). The file gives an agent brief context about a site and links to detailed material, and the file itself should be compact enough to fit in a model's context.
The format, in order:
1. An H1 with the project name. This is the only required element.
2. A blockquote with a short summary.
3. Any paragraphs and lists without headings: key facts, caveats, instructions.
4. Zero or more H2 sections, each with a list of links in the form `[name](url): note`.
5. An `## Optional` section: links an agent can skip when context is short.
The file sits at `/llms.txt` or under a subpath such as `/docs/llms.txt`; when several files match, the agent takes the most specific one. Separately, the specification suggests serving a clean Markdown version of each page at the same URL with a `.md` suffix (`page.html.md` or `page.md`, and `index.html.md` for URLs without a file name). For discovery, it suggests `rel="alternate" type="text/markdown"` for these versions and `rel="describedby"` for llms.txt itself, either in a `` tag or in an HTTP `Link:` header.
llms.txt is one author's proposal, not an IETF or W3C standard. No standardization process stands behind it, and none of the parties that might read the file has made any commitment.
| File | For whom | What it does | Required? |
|---|---|---|---|
| `robots.txt` | All crawlers, including AI bots | Allows or blocks crawling by user agent | De facto standard (RFC 9309), honored by major bots |
| `sitemap.xml` | Search crawlers | Lists all indexable URLs with dates | No, but it helps crawling |
| `llms.txt` | LLMs and agents | Gives a curated list of key pages with descriptions | No, support is unconfirmed |
| `.md` page versions | Agents reading a page | Serve the text without layout and scripts | No, a recommendation of the llms.txt spec |
The main difference from robots.txt: llms.txt allows and blocks nothing. If the goal is to let AI bots in or keep them out, robots.txt is the file to work with.
## Who reads llms.txt: what is confirmed and what is not
People make confident claims about llms.txt in both directions. Keep only primary sources, and the picture as of November 2026 looks like this.
| Claim | Status | Source |
|---|---|---|
| Google Search does not use llms.txt and does not require it | Confirmed | Google's guide to optimizing for generative search |
| Chrome Lighthouse checks that llms.txt is reachable | Confirmed | Lighthouse documentation |
| Most llms.txt files get no requests, and most requests do not come from AI platforms | Confirmed on the Ahrefs sample | Ahrefs study, May 2026 |
| OpenAI, Anthropic, Perplexity use llms.txt for answers or indexing | Unconfirmed | No official statements |
| Microsoft (Bing, Copilot) uses llms.txt | Unconfirmed | No statements found |
| Yandex (Search, Alice) uses llms.txt | Unconfirmed | No statements found |
| llms.txt increases citations in LLM answers | Unconfirmed | No studies with a control group |
"Unconfirmed" is more accurate here than "not read". The absence of a statement doesn't prove the file goes unused, and a few requests in logs do not prove it is used.
Companies publishing their own files proves nothing either. Anthropic has an [llms.txt for its docs](https://platform.claude.com/docs/llms.txt) that links to `llms-full.txt`, Perplexity has [docs.perplexity.ai/llms.txt](https://docs.perplexity.ai/llms.txt), and Yandex has an [llms.txt for the Webmaster API docs](https://yandex.ru/dev/webmaster/doc/ru/llms.txt). This shows that these companies find the file useful for their own documentation. It doesn't show that their crawlers read llms.txt on other sites.
### Google and Lighthouse: two different signals
In May 2026, Google published a [guide to optimizing for generative search](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide) ([announced on the Search Central Blog](https://developers.google.com/search/blog/2026/05/a-new-resource-for-optimizing) on May 15, 2026). The section on llms.txt and special markup says Google Search doesn't use these files and ignores them, they neither help nor hurt a site's visibility, and it's fine to maintain them for other systems. The page [AI features and your website](https://developers.google.com/search/docs/appearance/ai-features) adds a general principle: you don't need new machine-readable files, AI text files, or markup to appear in AI search features.
Google's John Mueller has addressed the topic twice. In June 2025, he wrote on Bluesky that no AI system currently uses llms.txt ([as reported by Search Engine Roundtable](https://www.seroundtable.com/google-ai-llms-txt-39607.html)). In June 2026, [according to Search Engine Journal's summary](https://www.searchenginejournal.com/googles-mueller-says-llms-txt-cant-help-llms-differentiate-sites/579304/) of a Search Off the Record podcast episode, he explained why the file doesn't help choose sources: the site owner writes llms.txt, so it can't tell one site from another. A narrow role is possible once an agent is already working with a site.
Chrome Lighthouse looks at llms.txt from another angle. Its [llms.txt audit](https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt) is documented in the agentic browsing section and checks one thing: whether the server returns an error when the file is requested. Chrome calls the file itself an "emerging convention". If there is no file and the server returns 404, the audit is marked Not Applicable, because the file is optional for now. This is a signal for browser agents, not a ranking factor.
Taken together: Search doesn't need the file, browser agents may find it useful, and a server error on `/llms.txt` is worse than having no file at all.
### Log data: what it shows and what it doesn't
The largest measurement is the [Ahrefs study](https://ahrefs.com/blog/llmstxt-study/), published on June 15, 2026. Ahrefs took 137,210 domains with traffic from its Web Analytics for May 2026.
| Metric | Value |
|---|---|
| Domains with llms.txt | about 28% of the sample (about 38K) |
| Files with zero requests in May | 97% |
| Share of requests from bots | 96% (4% from humans) |
| Named AI bots | 19.5% of requests |
| GPTBot / ClaudeBot / OAI-SearchBot | about 4.5% / 0.8% / 0.7% |
| SEO auditors | 21.7% (including 10.6% from Ahrefs' own crawlers) |
| Googlebot | about 900 requests, as part of normal crawling |
The caveats the authors give matter more than the numbers. A request for the file doesn't mean a model used its content. Only about 1.1K domains got any requests (about 22K requests), and the bot breakdown is built on that subset. The sample leans toward the technical sites of Ahrefs customers, so 28% is an upper-bound estimate. Files were not checked for compliance with the spec.
Individual observations from practitioners disagree. SEO specialist Ray Martinez showed logs in which OpenAI requested his llms.txt about every 15 minutes ([Search Engine Roundtable, July 2025](https://www.seroundtable.com/openai-crawling-Ilms-txt-files-39811.html)); other small-sample measurements see a handful of requests from AI bots, or none. Logs from one site say nothing about how a platform behaves overall.
For your own site, logs are the only honest way to answer "is anyone reading my file". The futurecraft.pro blog has no such measurement: the author has no logs of requests to `/llms.txt`, and Yandex Metrica and GA4 don't see bots. So there is nothing to support a claim that the file did anything for this site.
## Does your site need llms.txt?
The test is simple: the time spent maintaining the file should be smaller than the value to agents that actually work with the site. The file won't bring search traffic. For documentation it is almost always worth doing; for a blog it is more about tidiness than growth.
| Site type | Needed? | Why |
|---|---|---|
| Documentation, API, SDK | Yes | Coding agents and IDE assistants hit docs while they work; a table of contents with links to .md saves them [context](https://futurecraft.pro/blog/context-engineering-guide/). Generation is often built into the docs platform |
| SaaS with public help docs | Probably | The same agents, plus users asking assistants about the product; the cost is low |
| Blog, media, content site | Optional | No confirmed benefit; if content is in Markdown, the file and .md versions take an evening |
| Online store | Hardly | The catalog changes constantly, and a static list of dozens of links goes stale fast and doesn't cover the range |
| Landing site of a few pages | No | The page itself is enough for an agent |
On WordPress, [Yoast SEO](https://yoast.com/press/yoast-launches-the-first-llms-txt-generator-in-an-seo-plugin-ushering-in-a-new-era-of-ai-ready-content/) (including the free version) and [AIOSEO](https://aioseo.com/features/llms-txt/) both have a generator. Mintlify and GitBook, from the list on llmstxt.org, generate the file themselves, and VitePress and Docusaurus have plugins. Writing your own generator makes sense when content lives in a repository, as with a static site on Astro, Hugo, or Next.js.
## What a good llms.txt looks like
A minimal file per the spec for a hypothetical API documentation site:
```markdown
# Example API
> REST API documentation for the Example service: authentication, endpoints, SDKs for Python and TypeScript.
All examples use API v2. Version v1 has been disabled since 2026-01-01.
## Docs
- [Quickstart](https://example.com/docs/quickstart.md): API key and first request
- [Authentication](https://example.com/docs/auth.md): OAuth 2.0 and service tokens
- [API reference](https://example.com/docs/api.md): all endpoints with parameters
## Optional
- [Changelog](https://example.com/docs/changelog.md): API version history
```
Rules that follow from the spec and from practice:
- **Dozens of links.** The file is meant as a table of contents; the whole archive doesn't belong there. Everything secondary goes under `## Optional`.
- **A note for every link.** The agent picks a page by the note after the colon, so the note should say what is inside rather than repeat the title.
- **Free text before the first H2.** Parsers treat every H2 section as a list of links. Keep key facts and citation guidance as paragraphs after the blockquote.
- **Links to .md where available.** The spec asks for links to LLM-friendly content, such as Markdown versions of pages.
- **Live URLs only.** Every link should return 200, or the agent gets a dead end.
## How to generate llms.txt from content
When articles live in Markdown with frontmatter, a script can build llms.txt on every build. On the futurecraft.pro blog (Astro, about 130 articles in two locales), that job belongs to `scripts/generate-llms.mjs`. The data source is post frontmatter parsed with `gray-matter`: title, description, date, update date. Drafts with `draft: true` are filtered out, and articles are sorted by date.
The static part (H1, blockquote, key facts, citation guidance, main pages, the Optional section) lives in the script as template constants. Only the article lists and the article count are generated. Each article line is built like this:
```js
function buildArticlesSection(heading, articles) {
const lines = articles.map((a) => {
const desc = a.description ? ` ${a.description}` : '';
const updated = a.updated ? ` Updated: ${formatDate(a.updated)}.` : '';
return `- [${a.title}](${a.url}):${desc} Published: ${formatDate(a.date)}.${updated}`;
});
return `## ${heading}\n\n${lines.join('\n')}\n`;
}
```
The script runs in CI as a separate step before the build, so the deployed file never drifts from the published articles:
```yaml
- name: Regenerate llms.txt
run: node scripts/generate-llms.mjs
- name: Build
run: npm run build
```
The same approach works with any static site generator: Markdown and frontmatter are the single source of truth, and llms.txt, RSS, and the sitemap are all built from it. For more on Markdown content as the base of the whole publishing pipeline, see the article on the [AI Content Engine](https://futurecraft.pro/blog/ai-content-engine/).
### Checking links in CI
In August 2026, the blog's llms.txt turned out to contain links to `news` category pages in both locales. There were no published articles in that category, Astro doesn't build those routes, and the links returned 404. A human wouldn't notice; an agent following the table of contents would hit a broken URL. The same file had a literal placeholder `.md` in an example for agents.
There were two fixes. The generator now collects the set of categories that have published articles and drops links to empty ones. And the build's SEO audit reads `dist/llms.txt`, extracts every internal URL, and checks that the build has a page for each:
```js
const llms = fs.readFileSync(path.join(distDir, 'llms.txt'), 'utf8');
if (Buffer.byteLength(llms) > 50_000) {
errors.push(`llms.txt is ${Buffer.byteLength(llms)} bytes; keep it under 50 KB`);
}
const internalUrls = new Set(
[...llms.matchAll(/https:\/\/futurecraft\.pro[^)\s]+/g)].map((m) => m[0]),
);
for (const url of internalUrls) {
const pathname = localPath(url);
if (!pathname || !generatedTargetExists(pathname)) {
errors.push(`llms.txt: broken internal link ${url}`);
}
}
```
The size check came later, together with the cut in file size. The placeholder was replaced with the live URL `/blog/mcp-security-guide.md`.
### Size: from 111 KB to 31.5 KB
The blog's first llms.txt was a table of contents for the whole archive: 347 lines, 111,245 bytes (about 33 KB gzipped). It had the H1 and blockquote, key facts, main pages and categories in both locales, one line for each of roughly 130 English articles and as many Russian ones, citation guidance, and a section on Markdown versions.
That file departed from the spirit of the spec in two places. First, size: 111 KB is tens of thousands of tokens, while the spec asks for a file that fits in context. In effect it was a full index, closer to the `llms-full.txt` practice. Second, the key facts, citation guidance, and the Markdown note sat under H2 headings, while parsers expect a list of links in every H2.
The new version has 151 lines and 31,564 bytes (about 9.6 KB gzipped):
- **Preamble without headings:** the H1, blockquote, key facts, citation guidance, and the note about .md versions come as paragraphs before the first H2.
- **Main pages:** the site's main pages in both languages.
- **Key articles:** 22 key articles per locale. These are the core guides and the pages Copilot cites most often, according to Bing Webmaster Tools.
- **Latest articles:** the 10 latest articles per locale, excluding those already listed as key.
- **`## Optional`:** category pages, RSS feeds, and the sitemap. Through them an agent can reach the rest of the archive.
Two checks keep the file within these limits. The generator fails if an article on the key list isn't published in one of the locales, so nobody can forget to update the list after an article is taken down:
```js
function featuredArticles(articles, locale) {
const bySlug = new Map(articles.map((a) => [a.slug, a]));
const missing = FEATURED_SLUGS.filter((slug) => !bySlug.has(slug));
if (missing.length > 0) {
throw new Error(`FEATURED_SLUGS not published in ${locale}: ${missing.join(', ')}`);
}
return FEATURED_SLUGS.map((slug) => bySlug.get(slug));
}
```
The second check is a 50 KB limit in the SEO audit and in the production smoke test. The threshold leaves headroom above the current 31.5 KB so the file can't quietly grow back.
The cut has a cost: the key article list is maintained by hand, and deciding what counts as key stays with a human.
## .md versions of pages: what they give and how to serve them
If llms.txt is the table of contents, the .md version of a page is the content itself, without layout, menus, and scripts. The agent doesn't have to parse HTML, and fewer tokens go into context. Unlike llms.txt, .md versions leave an observable trace: in the AI Performance report in Bing Webmaster Tools, Copilot cites the URL `/blog/prompt-library-template.md` separately from the HTML version of the same article.
On the blog, .md versions have four parts.
**1. A static endpoint.** The files `src/pages/blog/[slug].md.ts` and `src/pages/ru/blog/[slug].md.ts` generate `/blog/.md` and `/ru/blog/.md` at build time for every published article in the matching locale. The body is the title, a blockquote with the description and an "author, date, canonical URL" line, then the article source:
```ts
export async function GET({ props }: APIContext) {
const entry = (props as { entry: CollectionEntry<'blog'> }).entry;
const slug = entry.slug.replace(/^en\//, '');
const url = `${SITE}/blog/${slug}/`;
const date = new Date(entry.data.date).toISOString().slice(0, 10);
const lines = [`# ${entry.data.title}`, ''];
if (entry.data.description) lines.push(`> ${entry.data.description}`);
lines.push(`> Author: Roman Belov · Published: ${date} · Source: ${url}`, '', entry.body.trim(), '');
return new Response(lines.join('\n'), {
headers: { 'Content-Type': 'text/markdown; charset=utf-8' },
});
}
```
The endpoint serves the source as is, so check that it contains no imports or MDX components. The blog's articles have none outside code blocks. Wherever the source has plain HTML, that HTML ends up in the .md.
**2. Negotiation on the Accept header.** The Cloudflare Pages Function `functions/_middleware.js` intercepts a `GET` to an article URL with `Accept: text/markdown` (or with the `?format=md` parameter) and returns the prebuilt `.md` at the canonical URL:
```js
const accept = context.request.headers.get('Accept') || '';
const wantsMd = accept.includes('text/markdown') || url.searchParams.get('format') === 'md';
const isArticle = /^\/(?:ru\/)?blog\/[^/]+\/?$/.test(url.pathname)
&& !url.pathname.includes('/category/') && !url.pathname.endsWith('.md');
if (context.request.method === 'GET' && wantsMd && isArticle && context.env.ASSETS) {
const mdUrl = new URL(url.pathname.replace(/\/$/, '') + '.md', url.origin);
const mdResp = await context.env.ASSETS.fetch(mdUrl);
if (mdResp.ok) {
return new Response(mdResp.body, {
status: 200,
headers: { 'Content-Type': 'text/markdown; charset=utf-8', 'Vary': 'Accept' },
});
}
}
```
`Vary: Accept` is mandatory here: without it, a CDN may cache the Markdown and serve it to a browser at the same URL.
**3. A link from the HTML.** The layout adds `` to the article's ``, as the spec suggests.
**4. Response headers.** In `public/_headers`:
```
/blog/*.md
Content-Type: text/markdown; charset=utf-8
X-Robots-Tag: noindex, follow
Link: ; rel="canonical"
/ru/blog/*.md
Content-Type: text/markdown; charset=utf-8
X-Robots-Tag: noindex, follow
Link: ; rel="canonical"
```
`noindex` keeps the .md out of the search index so it doesn't compete with the HTML page. `rel="canonical"` consolidates signals on the HTML version; it was added in October 2026, right after a .md URL showed up among the pages Copilot cites. You can check the headers in production with one command (the URL is the Markdown version of the article on [MCP security](https://futurecraft.pro/blog/mcp-security-guide/)):
```bash
curl -sI https://futurecraft.pro/blog/mcp-security-guide.md \
| grep -iE '^(content-type|x-robots-tag|link):'
```
The endpoint rewrites links and images inside the .md to absolute URLs (`https://futurecraft.pro/blog/...`) and leaves code blocks alone. In the article source they are relative, and an agent that gets the file without a base URL could not follow them. One caveat remains: llms.txt itself links to the HTML URLs of articles, even though the spec advises linking to LLM-friendly content such as .md. The file's preamble explains how to get Markdown: the `.md` suffix or the `Accept` header. The case for HTML links: the assistant's answer should carry the page's canonical URL, not a service one. The case against: the agent has to take an extra step. Both options work.
## llms.txt and GEO: what actually affects citations
GEO (generative engine optimization) is the work of getting AI assistants to use a site's pages as sources for answers. llms.txt is often presented as the main GEO tool, although nothing confirms this. Things backed by documented mechanisms matter more.
**robots.txt for AI bots.** This is what decides whether a crawler gets onto the site. The blog's robots.txt explicitly allows bots from OpenAI, Anthropic, and Perplexity (OAI-SearchBot, GPTBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User), as well as the Google-Extended and Applebot-Extended tokens. CCBot, Bytespider, and meta-externalagent are blocked: the file labels them as bulk training crawlers. A mistake in robots.txt can remove a site from answers; a mistake in llms.txt has no confirmed effect on anything.
**Accessible HTML.** The text should be readable from HTML without running JavaScript. In the same podcast episode, according to SEJ's summary, Mueller recommends plain HTML pages and internal links for discovery.
**Page structure.** A definition in the first paragraph, comparison tables, short self-contained paragraphs under headings, and an FAQ with complete answers.
**Regular indexing.** Copilot finds material through the Bing index, so pages need to get into it. IndexNow can speed up indexing of new and updated pages.
There is an observation from this site, but it doesn't prove causation. From August 6 to November 5, 2026, the AI Performance report in Bing Webmaster Tools showed 3,536 citations of 81 blog pages in Copilot and Microsoft partners, and citations grew month over month: 972 in September, 2,028 in October. llms.txt, .md versions, and an open robots.txt were in place for the entire period, and there is no control group, so the growth in citations can't be attributed to llms.txt. The pages cited most often are long guides with definitions and tables. The data is broken down in the case study [GEO in practice: 3,500 citations in Copilot](https://futurecraft.pro/blog/geo-copilot-case/).
## What not to expect from llms.txt and how to measure the effect
What you won't get:
- **Higher Google rankings.** Google says plainly that Search ignores the file.
- **Guaranteed citations.** No AI platform has promised to consider llms.txt when choosing sources.
- **Content protection.** The file blocks nothing. To keep a site out of training, you need robots.txt, and for bots that ignore it, WAF rules.
- **Faster indexing.** That is the job of the sitemap, IndexNow, and search engine APIs.
- **Protection from hallucinations or a guaranteed lift.** Claims like "llms.txt prevents hallucinations" or "raises the chance of citation by 20–25%" appear in Russian-language guides with no source or methodology.
What you can measure:
| What we measure | Tool | What it shows |
|---|---|---|
| Who requests `/llms.txt` | Web server or CDN logs, filtered by path and user agent | Which bots fetch the file and how often. Won't show whether the file was used in an answer |
| Who requests .md versions | The same logs, by the `*.md` mask | Whether agents use the Markdown versions |
| Citations in Copilot | Bing Webmaster Tools, AI Performance | Citations by page and grounding queries. Cited .md URLs appear separately |
| File availability | Chrome Lighthouse, llms.txt audit | Whether the server returns an error |
| Broken links in the file | A check in CI | Whether every URL in llms.txt exists |
Keep logs with dates before and after each change. Without them, any talk about the effect of llms.txt stays at the level of opinion, and that applies to every site, this one included.
## llms.txt implementation checklist
1. Decide whether you need the file: documentation and APIs, yes; a blog, optional; a store or landing page, probably not.
2. Check robots.txt: the AI bots you want should not be blocked.
3. Build the file per the spec: H1, blockquote, free text without headings, H2 sections with links, `## Optional`.
4. Limit the size: dozens of key links, with the rest reachable through category pages, RSS, and the sitemap under `## Optional`.
5. Give every link a note that says what is on the page.
6. Generate the file from content on every build instead of editing it by hand.
7. Check in CI that every link in the file points to an existing page, and cap the size.
8. If your content is in Markdown, serve `.md` versions of pages with `X-Robots-Tag: noindex` and `Link: rel=canonical` pointing to the HTML.
9. Add `` to pages and, optionally, negotiation on `Accept: text/markdown` with `Vary: Accept`.
10. Set up measurement: logs of requests to `/llms.txt` and `.md`, AI Performance in Bing Webmaster Tools, the Lighthouse audit.
## Sources
- [llmstxt.org](https://llmstxt.org/): the llms.txt specification (Jeremy Howard, 2024).
- [Google Search Central: optimizing for generative search](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide) and [the guide's announcement, May 15, 2026](https://developers.google.com/search/blog/2026/05/a-new-resource-for-optimizing).
- [Google Search Central: AI features and your website](https://developers.google.com/search/docs/appearance/ai-features).
- [Chrome Lighthouse: llms.txt audit](https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt).
- [Ahrefs: llms.txt study, June 2026](https://ahrefs.com/blog/llmstxt-study/).
- [Search Engine Journal: Mueller on llms.txt, June 2026](https://www.searchenginejournal.com/googles-mueller-says-llms-txt-cant-help-llms-differentiate-sites/579304/).
- [Search Engine Roundtable: Mueller on Bluesky, June 2025](https://www.seroundtable.com/google-ai-llms-txt-39607.html).
- [Yoast SEO: llms.txt generator](https://yoast.com/press/yoast-launches-the-first-llms-txt-generator-in-an-seo-plugin-ushering-in-a-new-era-of-ai-ready-content/) and [AIOSEO: LLMs.txt Generator](https://aioseo.com/features/llms-txt/).