llms.txt: Do You Need It and How to Build It Right

llms.txt is a Markdown file in the site root with a short description of the project and a list of key pages for language models and agents. Jeremy Howard proposed the format in September 2024. Since then, Yoast SEO and AIOSEO have added llms.txt generators, Anthropic, Perplexity, and Yandex publish the file for their docs, and Chrome Lighthouse has a dedicated audit. Yet no major AI platform has confirmed that it reads the file, and Google says outright that Search ignores it.

According to Ahrefs, 97% of llms.txt files got zero requests in May 2026. The other 3% were fetched mostly by SEO auditors and other bots: named AI bots accounted for 19.5% of requests.

This article covers what is confirmed about llms.txt and what is not, who actually needs the file, how to generate it from content and check it in CI, why .md versions of pages belong next to it, and how to measure the effect. The examples come from the futurecraft.pro blog: the generator, an incident with broken links, and cutting the file from 111 KB to 31.5 KB.

What llms.txt is and how it differs from robots.txt and sitemap.xml

The specification lives at llmstxt.org. The file gives an agent brief context about a site and links to detailed material, and the file itself should be compact enough to fit in a model’s context.

The format, in order:

1. An H1 with the project name. This is the only required element.
2. A blockquote with a short summary.
3. Any paragraphs and lists without headings: key facts, caveats, instructions.
4. Zero or more H2 sections, each with a list of links in the form [name](url): note.
5. An ## Optional section: links an agent can skip when context is short.

The file sits at /llms.txt or under a subpath such as /docs/llms.txt; when several files match, the agent takes the most specific one. Separately, the specification suggests serving a clean Markdown version of each page at the same URL with a .md suffix (page.html.md or page.md, and index.html.md for URLs without a file name). For discovery, it suggests rel="alternate" type="text/markdown" for these versions and rel="describedby" for llms.txt itself, either in a <link> tag or in an HTTP Link: header.

llms.txt is one author’s proposal, not an IETF or W3C standard. No standardization process stands behind it, and none of the parties that might read the file has made any commitment.

The main difference from robots.txt: llms.txt allows and blocks nothing. If the goal is to let AI bots in or keep them out, robots.txt is the file to work with.

Who reads llms.txt: what is confirmed and what is not

People make confident claims about llms.txt in both directions. Keep only primary sources, and the picture as of November 2026 looks like this.

“Unconfirmed” is more accurate here than “not read”. The absence of a statement doesn’t prove the file goes unused, and a few requests in logs do not prove it is used.

Companies publishing their own files proves nothing either. Anthropic has an llms.txt for its docs that links to llms-full.txt, Perplexity has docs.perplexity.ai/llms.txt, and Yandex has an llms.txt for the Webmaster API docs. This shows that these companies find the file useful for their own documentation. It doesn’t show that their crawlers read llms.txt on other sites.

Google and Lighthouse: two different signals

In May 2026, Google published a guide to optimizing for generative search (announced on the Search Central Blog on May 15, 2026). The section on llms.txt and special markup says Google Search doesn’t use these files and ignores them, they neither help nor hurt a site’s visibility, and it’s fine to maintain them for other systems. The page AI features and your website adds a general principle: you don’t need new machine-readable files, AI text files, or markup to appear in AI search features.

Google’s John Mueller has addressed the topic twice. In June 2025, he wrote on Bluesky that no AI system currently uses llms.txt (as reported by Search Engine Roundtable). In June 2026, according to Search Engine Journal’s summary of a Search Off the Record podcast episode, he explained why the file doesn’t help choose sources: the site owner writes llms.txt, so it can’t tell one site from another. A narrow role is possible once an agent is already working with a site.

Chrome Lighthouse looks at llms.txt from another angle. Its llms.txt audit is documented in the agentic browsing section and checks one thing: whether the server returns an error when the file is requested. Chrome calls the file itself an “emerging convention”. If there is no file and the server returns 404, the audit is marked Not Applicable, because the file is optional for now. This is a signal for browser agents, not a ranking factor.

Taken together: Search doesn’t need the file, browser agents may find it useful, and a server error on /llms.txt is worse than having no file at all.

Log data: what it shows and what it doesn’t

The largest measurement is the Ahrefs study, published on June 15, 2026. Ahrefs took 137,210 domains with traffic from its Web Analytics for May 2026.

The caveats the authors give matter more than the numbers. A request for the file doesn’t mean a model used its content. Only about 1.1K domains got any requests (about 22K requests), and the bot breakdown is built on that subset. The sample leans toward the technical sites of Ahrefs customers, so 28% is an upper-bound estimate. Files were not checked for compliance with the spec.

Individual observations from practitioners disagree. SEO specialist Ray Martinez showed logs in which OpenAI requested his llms.txt about every 15 minutes (Search Engine Roundtable, July 2025); other small-sample measurements see a handful of requests from AI bots, or none. Logs from one site say nothing about how a platform behaves overall.

For your own site, logs are the only honest way to answer “is anyone reading my file”. The futurecraft.pro blog has no such measurement: the author has no logs of requests to /llms.txt, and Yandex Metrica and GA4 don’t see bots. So there is nothing to support a claim that the file did anything for this site.

Does your site need llms.txt?

The test is simple: the time spent maintaining the file should be smaller than the value to agents that actually work with the site. The file won’t bring search traffic. For documentation it is almost always worth doing; for a blog it is more about tidiness than growth.

On WordPress, Yoast SEO (including the free version) and AIOSEO both have a generator. Mintlify and GitBook, from the list on llmstxt.org, generate the file themselves, and VitePress and Docusaurus have plugins. Writing your own generator makes sense when content lives in a repository, as with a static site on Astro, Hugo, or Next.js.

What a good llms.txt looks like

A minimal file per the spec for a hypothetical API documentation site:

# Example API

> REST API documentation for the Example service: authentication, endpoints, SDKs for Python and TypeScript.

All examples use API v2. Version v1 has been disabled since 2026-01-01.

## Docs

- [Quickstart](https://example.com/docs/quickstart.md): API key and first request
- [Authentication](https://example.com/docs/auth.md): OAuth 2.0 and service tokens
- [API reference](https://example.com/docs/api.md): all endpoints with parameters

## Optional

- [Changelog](https://example.com/docs/changelog.md): API version history

Rules that follow from the spec and from practice:

• Dozens of links. The file is meant as a table of contents; the whole archive doesn’t belong there. Everything secondary goes under ## Optional.
• A note for every link. The agent picks a page by the note after the colon, so the note should say what is inside rather than repeat the title.
• Free text before the first H2. Parsers treat every H2 section as a list of links. Keep key facts and citation guidance as paragraphs after the blockquote.
• Links to .md where available. The spec asks for links to LLM-friendly content, such as Markdown versions of pages.
• Live URLs only. Every link should return 200, or the agent gets a dead end.

How to generate llms.txt from content

When articles live in Markdown with frontmatter, a script can build llms.txt on every build. On the futurecraft.pro blog (Astro, about 130 articles in two locales), that job belongs to scripts/generate-llms.mjs. The data source is post frontmatter parsed with gray-matter: title, description, date, update date. Drafts with draft: true are filtered out, and articles are sorted by date.

The static part (H1, blockquote, key facts, citation guidance, main pages, the Optional section) lives in the script as template constants. Only the article lists and the article count are generated. Each article line is built like this:

function buildArticlesSection(heading, articles) {
  const lines = articles.map((a) => {
    const desc = a.description ? ` ${a.description}` : '';
    const updated = a.updated ? ` Updated: ${formatDate(a.updated)}.` : '';
    return `- [${a.title}](${a.url}):${desc} Published: ${formatDate(a.date)}.${updated}`;
  });
  return `## ${heading}\n\n${lines.join('\n')}\n`;
}

The script runs in CI as a separate step before the build, so the deployed file never drifts from the published articles:

- name: Regenerate llms.txt
  run: node scripts/generate-llms.mjs

- name: Build
  run: npm run build

The same approach works with any static site generator: Markdown and frontmatter are the single source of truth, and llms.txt, RSS, and the sitemap are all built from it. For more on Markdown content as the base of the whole publishing pipeline, see the article on the AI Content Engine.

Checking links in CI

In August 2026, the blog’s llms.txt turned out to contain links to news category pages in both locales. There were no published articles in that category, Astro doesn’t build those routes, and the links returned 404. A human wouldn’t notice; an agent following the table of contents would hit a broken URL. The same file had a literal placeholder <slug>.md in an example for agents.

There were two fixes. The generator now collects the set of categories that have published articles and drops links to empty ones. And the build’s SEO audit reads dist/llms.txt, extracts every internal URL, and checks that the build has a page for each:

const llms = fs.readFileSync(path.join(distDir, 'llms.txt'), 'utf8');
if (Buffer.byteLength(llms) > 50_000) {
  errors.push(`llms.txt is ${Buffer.byteLength(llms)} bytes; keep it under 50 KB`);
}
const internalUrls = new Set(
  [...llms.matchAll(/https:\/\/futurecraft\.pro[^)\s]+/g)].map((m) => m[0]),
);
for (const url of internalUrls) {
  const pathname = localPath(url);
  if (!pathname || !generatedTargetExists(pathname)) {
    errors.push(`llms.txt: broken internal link ${url}`);
  }
}

The size check came later, together with the cut in file size. The placeholder was replaced with the live URL /blog/mcp-security-guide.md.

Size: from 111 KB to 31.5 KB

The blog’s first llms.txt was a table of contents for the whole archive: 347 lines, 111,245 bytes (about 33 KB gzipped). It had the H1 and blockquote, key facts, main pages and categories in both locales, one line for each of roughly 130 English articles and as many Russian ones, citation guidance, and a section on Markdown versions.

That file departed from the spirit of the spec in two places. First, size: 111 KB is tens of thousands of tokens, while the spec asks for a file that fits in context. In effect it was a full index, closer to the llms-full.txt practice. Second, the key facts, citation guidance, and the Markdown note sat under H2 headings, while parsers expect a list of links in every H2.

The new version has 151 lines and 31,564 bytes (about 9.6 KB gzipped):

• Preamble without headings: the H1, blockquote, key facts, citation guidance, and the note about .md versions come as paragraphs before the first H2.
• Main pages: the site’s main pages in both languages.
• Key articles: 22 key articles per locale. These are the core guides and the pages Copilot cites most often, according to Bing Webmaster Tools.
• Latest articles: the 10 latest articles per locale, excluding those already listed as key.
• ## Optional: category pages, RSS feeds, and the sitemap. Through them an agent can reach the rest of the archive.

Two checks keep the file within these limits. The generator fails if an article on the key list isn’t published in one of the locales, so nobody can forget to update the list after an article is taken down:

function featuredArticles(articles, locale) {
  const bySlug = new Map(articles.map((a) => [a.slug, a]));
  const missing = FEATURED_SLUGS.filter((slug) => !bySlug.has(slug));
  if (missing.length > 0) {
    throw new Error(`FEATURED_SLUGS not published in ${locale}: ${missing.join(', ')}`);
  }
  return FEATURED_SLUGS.map((slug) => bySlug.get(slug));
}

The second check is a 50 KB limit in the SEO audit and in the production smoke test. The threshold leaves headroom above the current 31.5 KB so the file can’t quietly grow back.

The cut has a cost: the key article list is maintained by hand, and deciding what counts as key stays with a human.

.md versions of pages: what they give and how to serve them

If llms.txt is the table of contents, the .md version of a page is the content itself, without layout, menus, and scripts. The agent doesn’t have to parse HTML, and fewer tokens go into context. Unlike llms.txt, .md versions leave an observable trace: in the AI Performance report in Bing Webmaster Tools, Copilot cites the URL /blog/prompt-library-template.md separately from the HTML version of the same article.

On the blog, .md versions have four parts.

1. A static endpoint. The files src/pages/blog/[slug].md.ts and src/pages/ru/blog/[slug].md.ts generate /blog/<slug>.md and /ru/blog/<slug>.md at build time for every published article in the matching locale. The body is the title, a blockquote with the description and an “author, date, canonical URL” line, then the article source:

export async function GET({ props }: APIContext) {
  const entry = (props as { entry: CollectionEntry<'blog'> }).entry;
  const slug = entry.slug.replace(/^en\//, '');
  const url = `${SITE}/blog/${slug}/`;
  const date = new Date(entry.data.date).toISOString().slice(0, 10);

  const lines = [`# ${entry.data.title}`, ''];
  if (entry.data.description) lines.push(`> ${entry.data.description}`);
  lines.push(`> Author: Roman Belov · Published: ${date} · Source: ${url}`, '', entry.body.trim(), '');

  return new Response(lines.join('\n'), {
    headers: { 'Content-Type': 'text/markdown; charset=utf-8' },
  });
}

The endpoint serves the source as is, so check that it contains no imports or MDX components. The blog’s articles have none outside code blocks. Wherever the source has plain HTML, that HTML ends up in the .md.

2. Negotiation on the Accept header. The Cloudflare Pages Function functions/_middleware.js intercepts a GET to an article URL with Accept: text/markdown (or with the ?format=md parameter) and returns the prebuilt .md at the canonical URL:

const accept = context.request.headers.get('Accept') || '';
const wantsMd = accept.includes('text/markdown') || url.searchParams.get('format') === 'md';
const isArticle = /^\/(?:ru\/)?blog\/[^/]+\/?$/.test(url.pathname)
  && !url.pathname.includes('/category/') && !url.pathname.endsWith('.md');

if (context.request.method === 'GET' && wantsMd && isArticle && context.env.ASSETS) {
  const mdUrl = new URL(url.pathname.replace(/\/$/, '') + '.md', url.origin);
  const mdResp = await context.env.ASSETS.fetch(mdUrl);
  if (mdResp.ok) {
    return new Response(mdResp.body, {
      status: 200,
      headers: { 'Content-Type': 'text/markdown; charset=utf-8', 'Vary': 'Accept' },
    });
  }
}

Vary: Accept is mandatory here: without it, a CDN may cache the Markdown and serve it to a browser at the same URL.

3. A link from the HTML. The layout adds <link rel="alternate" type="text/markdown" href="/blog/<slug>.md"> to the article’s <head>, as the spec suggests.

4. Response headers. In public/_headers:

/blog/*.md
  Content-Type: text/markdown; charset=utf-8
  X-Robots-Tag: noindex, follow
  Link: <https://futurecraft.pro/blog/:splat/>; rel="canonical"

/ru/blog/*.md
  Content-Type: text/markdown; charset=utf-8
  X-Robots-Tag: noindex, follow
  Link: <https://futurecraft.pro/ru/blog/:splat/>; rel="canonical"

noindex keeps the .md out of the search index so it doesn’t compete with the HTML page. rel="canonical" consolidates signals on the HTML version; it was added in October 2026, right after a .md URL showed up among the pages Copilot cites. You can check the headers in production with one command (the URL is the Markdown version of the article on MCP security):

curl -sI https://futurecraft.pro/blog/mcp-security-guide.md \
  | grep -iE '^(content-type|x-robots-tag|link):'

The endpoint rewrites links and images inside the .md to absolute URLs (https://futurecraft.pro/blog/...) and leaves code blocks alone. In the article source they are relative, and an agent that gets the file without a base URL could not follow them. One caveat remains: llms.txt itself links to the HTML URLs of articles, even though the spec advises linking to LLM-friendly content such as .md. The file’s preamble explains how to get Markdown: the .md suffix or the Accept header. The case for HTML links: the assistant’s answer should carry the page’s canonical URL, not a service one. The case against: the agent has to take an extra step. Both options work.

llms.txt and GEO: what actually affects citations

GEO (generative engine optimization) is the work of getting AI assistants to use a site’s pages as sources for answers. llms.txt is often presented as the main GEO tool, although nothing confirms this. Things backed by documented mechanisms matter more.

robots.txt for AI bots. This is what decides whether a crawler gets onto the site. The blog’s robots.txt explicitly allows bots from OpenAI, Anthropic, and Perplexity (OAI-SearchBot, GPTBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User), as well as the Google-Extended and Applebot-Extended tokens. CCBot, Bytespider, and meta-externalagent are blocked: the file labels them as bulk training crawlers. A mistake in robots.txt can remove a site from answers; a mistake in llms.txt has no confirmed effect on anything.

Accessible HTML. The text should be readable from HTML without running JavaScript. In the same podcast episode, according to SEJ’s summary, Mueller recommends plain HTML pages and internal links for discovery.

Page structure. A definition in the first paragraph, comparison tables, short self-contained paragraphs under headings, and an FAQ with complete answers.

Regular indexing. Copilot finds material through the Bing index, so pages need to get into it. IndexNow can speed up indexing of new and updated pages.

There is an observation from this site, but it doesn’t prove causation. From August 6 to November 5, 2026, the AI Performance report in Bing Webmaster Tools showed 3,536 citations of 81 blog pages in Copilot and Microsoft partners, and citations grew month over month: 972 in September, 2,028 in October. llms.txt, .md versions, and an open robots.txt were in place for the entire period, and there is no control group, so the growth in citations can’t be attributed to llms.txt. The pages cited most often are long guides with definitions and tables. The data is broken down in the case study GEO in practice: 3,500 citations in Copilot.

What not to expect from llms.txt and how to measure the effect

What you won’t get:

• Higher Google rankings. Google says plainly that Search ignores the file.
• Guaranteed citations. No AI platform has promised to consider llms.txt when choosing sources.
• Content protection. The file blocks nothing. To keep a site out of training, you need robots.txt, and for bots that ignore it, WAF rules.
• Faster indexing. That is the job of the sitemap, IndexNow, and search engine APIs.
• Protection from hallucinations or a guaranteed lift. Claims like “llms.txt prevents hallucinations” or “raises the chance of citation by 20–25%” appear in Russian-language guides with no source or methodology.

What you can measure:

Keep logs with dates before and after each change. Without them, any talk about the effect of llms.txt stays at the level of opinion, and that applies to every site, this one included.

llms.txt implementation checklist

1. Decide whether you need the file: documentation and APIs, yes; a blog, optional; a store or landing page, probably not.
2. Check robots.txt: the AI bots you want should not be blocked.
3. Build the file per the spec: H1, blockquote, free text without headings, H2 sections with links, ## Optional.
4. Limit the size: dozens of key links, with the rest reachable through category pages, RSS, and the sitemap under ## Optional.
5. Give every link a note that says what is on the page.
6. Generate the file from content on every build instead of editing it by hand.
7. Check in CI that every link in the file points to an existing page, and cap the size.
8. If your content is in Markdown, serve .md versions of pages with X-Robots-Tag: noindex and Link: rel=canonical pointing to the HTML.
9. Add <link rel="alternate" type="text/markdown"> to pages and, optionally, negotiation on Accept: text/markdown with Vary: Accept.
10. Set up measurement: logs of requests to /llms.txt and .md, AI Performance in Bing Webmaster Tools, the Lighthouse audit.

Sources

• llmstxt.org: the llms.txt specification (Jeremy Howard, 2024).
• Google Search Central: optimizing for generative search and the guide’s announcement, May 15, 2026.
• Google Search Central: AI features and your website.
• Chrome Lighthouse: llms.txt audit.
• Ahrefs: llms.txt study, June 2026.
• Search Engine Journal: Mueller on llms.txt, June 2026.
• Search Engine Roundtable: Mueller on Bluesky, June 2025.
• Yoast SEO: llms.txt generator and AIOSEO: LLMs.txt Generator.