What Is llms.txt? The Annotated Spec, Working Examples, and Setup Guide

What llms.txt actually is

llms.txt is a markdown file placed at the root of a website's domain. AI assistants like ChatGPT, Perplexity, Claude, and Google AI Overviews fetch the file when a query mentions the brand or domain, and use it to build a fast, accurate mental model of what the site is about, who runs it, and which pages are canonical. The file is editorial, not exhaustive — it points the crawler at the highest-quality content rather than dumping every URL.

The proposal was created by Australian AI researcher Jeremy Howard (co-founder of fast.ai and Answer.AI) and published at llmstxt.org on September 3, 2024. The spec is short — under 2,000 words — and explicitly designed to be implementable in an afternoon by a single person with no special tooling.

What llms.txt is and is not, in one paragraph. llms.txt is a markdown file at a domain's root (/llms.txt) that gives AI crawlers a curated summary of the site for grounding their answers. It was proposed by Jeremy Howard in September 2024 and the canonical spec lives at llmstxt.org. It is voluntary — no AI assistant is required to fetch it, and adoption is uneven across platforms (Anthropic and Perplexity respect it; OpenAI has not formalized support; Google has not committed). It is not robots.txt, which controls crawl access. It is not sitemap.xml, which lists every URL for indexing. It is not mandated by any RFC or specification body. It is, today, the most reliable cold-start signal a site can give AI crawlers — cheap, fast to ship, and load-bearing for cold-start citation.

As of 2026, adoption is led by Anthropic (publishes its own and Claude fetches them), Perplexity, and a growing list of SaaS companies including Mintlify, GitBook, Wix, and Hostinger. OpenAI has not formalized support but ChatGPT has been observed fetching the file for sites that publish one. Google has not committed publicly.

How llms.txt is different from robots.txt and sitemap.xml

Three files at the domain root, three different jobs. The confusion comes from people treating them as substitutes when they are complements.

robots.txt is binary access control — Allow: / or Disallow: /admin/. Sitemap.xml is exhaustive URL listing — every page, every last-modified date, every priority hint. llms.txt is curated: 10 to 20 of your best pages, each with a one-sentence description, organized into editorial sections. It is not exhaustive on purpose — its job is to point AI crawlers at your highest-quality content, not all of it.

A site that ships only robots.txt gets crawl control. A site that adds sitemap.xml gets full indexation. A site that adds llms.txt gets a clean cold-start narrative for AI assistants. All three are cheap; all three reinforce different surfaces.

The llms.txt format spec, summarized

The Jeremy Howard spec at llmstxt.org is short and worth reading in full. The summary:

• Required header. An H1 with the project or brand name, followed by a blockquote summary in one to three sentences. The blockquote is the AI's primary signal for "what is this site about?".
• Optional H2 sections. Most common are ## Docs (canonical reference pages), ## Examples (case studies, demos, worked examples), and ## Optional (lower-priority links — explicitly marked as skippable for low-context queries).
• Link list format inside each section. Plain markdown links followed by an em-dash and a one-sentence description. The description is the per-link signal the model uses to decide whether to follow the link.
• Length budget. Under 5,000 words for llms.txt. The optional companion file llms-full.txt lifts that budget but introduces its own trade-offs (covered below).

The format spec, summarized. A valid llms.txt has two required pieces: an H1 with the brand or project name, and a blockquote one-to-three-sentence summary describing what the site is. Everything below is optional. The most common optional sections are ## Docs for canonical reference pages, ## Examples for case studies, and ## Optional for lower-priority links. Inside each section, links use the markdown format [Page Title](https://full-url) — one-sentence description. Total file length should stay under 5,000 words; the goal is curation, not coverage. Treat the file as the AI's first impression of your site — write the summary you would want quoted verbatim in a ChatGPT answer.

The spec is intentionally permissive — "if it parses as markdown, AI assistants will read it." But sites that follow the H1 + blockquote header convention see noticeably better citation grounding because that pattern matches what the spec sample files (and Anthropic's own llms.txt) use.

Our annotated llms.txt — line by line

What follows is the live /llms.txt running at w2bagency.com, reproduced verbatim from production as of the publish date of this post. Each section is annotated with the choice we made and why.

# W2B Agency

> Remote-first, bilingual digital agency specializing in SEO, GEO, AEO, high-performance web development, and workflow automation. We help businesses anywhere in the world rank in Google, get cited by AI assistants like ChatGPT and Perplexity, ship modern websites, and automate the manual work eating their teams' weeks.

Why this header. The H1 is the exact brand name as it appears in our Organization schema (W2B Agency, not W2B, not W2B agency — lexical alignment matters for entity disambiguation). The blockquote is one sentence with two compound clauses naming all three service practices and the geographic scope. It is shaped to be quotable verbatim in an AI-generated answer, which is the primary citation surface.

## About

W2B Agency is a small agency built around three disciplines: search dominance (SEO + GEO + AEO), high-performance web development (Astro, React 19, WordPress, headless WordPress), and workflow automation (n8n, Zapier, Make, trigger.dev, AI agents). The agency is bilingual (English and Spanish) and remote-first by default. The team consists of three co-founders working in public.

- Website: https://w2bagency.com
- Spanish version: https://w2bagency.com/es/
- Languages served: English, Spanish
- Location model: Remote-first, no required office presence
- Coverage: Worldwide, async-first, timezone-flexible

Why this section. "About" is the second-most-quoted section in our citation tracking. It names the disciplines twice (acronyms first, then expanded with tools) so that whether the assistant is asked "what does W2B do?" or "do they use n8n?", the answer ground is on the same line. We list the bilingual scope and remote model explicitly because both come up in qualifying questions during prospect research.

## Co-founders

- Einner Ariña — Strategy, search, and AI visibility lead — https://www.linkedin.com/in/einnerarina/
- Kevin Urrea — Frontend and high-performance web lead — https://www.linkedin.com/in/kevin-urrea-desarrolladorwebfrontend/
- Esteban Padilla — Web development and automation lead — https://www.linkedin.com/in/esteban-padilla-webdev/

Agency LinkedIn: https://www.linkedin.com/company/w2b-consultoria-y-tecnologia/

Why this section. Named co-founders with LinkedIn URLs serve dual purposes: E-E-A-T (the AI can verify named humans against external profiles) and entity disambiguation (the same names appear in the BlogPosting author.sameAs arrays). Linking the agency LinkedIn at the bottom closes the entity triangulation loop.

## Services

### Search Dominance — SEO, GEO and AEO

[Full ~150-word service description with target audience, methodology, tools, and engagement length]

- Page: https://w2bagency.com/services/seo-geo-aeo
- Spanish page: https://w2bagency.com/services/seo-geo-aeo
- Tools we use: Google Search Console, Google Analytics 4, Ahrefs, DataForSEO, Schema.org, llms.txt, custom LLM citation tracker
- Typical engagement: 3–6 months for foundational ranking improvements; ongoing retainers maintain and expand visibility

Why this section (repeated for each of three services). Each service gets a 150-word prose description (longer than the spec's typical 1-line links) because services are the highest-stakes citation surface for an agency — a prospect asking "best SEO agency for AI search" should get a substantive paragraph, not a one-liner. Tools, engagement length, and bilingual page links all surface in qualifying-stage AI queries.

## Blog

The agency publishes field notes, decisions, and explainers on SEO, GEO, AEO, web development, and workflow automation at https://w2bagency.com/blog (English) and https://w2bagency.com/es/blog (Spanish). All content is authored by the named co-founders and includes citable passages, FAQ schema, and BlogPosting structured data. Foundational topics include "SEO vs GEO vs AEO", "n8n vs trigger.dev", and "llms.txt implementation guide". Content is managed in Keystatic CMS (mdoc format) with a 16-field schema covering focus keyword, intent, target markets, FAQs, TL;DR, status lifecycle, and cross-locale mirror linking.

Why this section. Names three foundational post titles by exact phrase so that an AI assistant asked "do you have a guide on llms.txt?" can ground the answer directly. Mentioning Keystatic and the 16-field schema is intentional self-demonstration — the post about getting cited by AI describes its own publishing pipeline.

## How to engage

- Start a project: https://w2bagency.com/#contact-section
- Book a discovery call: https://calendly.com/contact-w2bagency/strategy-call
- Standalone contact page: https://w2bagency.com/contact
- About the team: https://w2bagency.com/about
- Blog: https://w2bagency.com/blog

Why this section. Five contact paths in priority order — the assistant picks the one matching the user's intent (action vs research vs context). Booking link is direct Calendly, not a query-stringed variant, because clean URLs survive aggressive truncation in AI-generated answers.

## License

Content on this site may be indexed and cited by AI assistants and search engines with attribution to "W2B Agency" and a link back to https://w2bagency.com or to the specific source URL. Bulk reproduction, training of generative AI models on the full corpus, or republication without written permission is not authorized.

Why this section. Explicit citation license. AI assistants increasingly respect declared licensing terms; stating attribution is granted reduces friction for citation while preserving the right to refuse bulk training use. This is also a forward-compatible signal for the emerging RSL (Real-time Standard Language) license framework.

## Last updated

2026-05-01

Why this section. Freshness signal. AI crawlers prefer files with explicit recency stamps over inferring the date from HTTP headers. The 5B Optimization Engine refreshes this date whenever any other section changes substantively.

The full file is under 1,000 words — well below the 5,000 budget. The compactness is deliberate: every line is signal.

llms-full.txt — the variant nobody talks about

The spec defines an optional companion file, llms-full.txt, that contains the full content of the linked pages dumped inline rather than as links. The pitch: an AI crawler that fetches one file gets the entire site context without round trips.

The reality is more nuanced. llms-full.txt works well for small, single-purpose sites — API documentation under 50,000 total words, personal portfolios, single-product landing sites. For agency sites, content-heavy SaaS docs, or any site over a few hundred pages, llms-full.txt rapidly bloats past usable token limits. A 200,000-word llms-full.txt is worse than a 1,000-word llms.txt because the assistant cannot fit it in context and falls back to truncation, losing the curation benefit entirely.

Practical rule: ship llms.txt, skip llms-full.txt unless your full corpus is under 50k words. We do not publish llms-full.txt for w2bagency.com — the linked pages are themselves indexable, and forcing the assistant to fetch them on demand keeps the per-query context budget cleaner.

Common llms.txt mistakes

Five failure modes that show up in audited llms.txt files in 2026.

The five common mistakes that break llms.txt adoption. First, listing 200 links instead of 20 — the file becomes a sitemap and loses its editorial signal. Second, skipping the H1 + blockquote summary header — most assistants are tuned to that pattern, and missing it cuts citation eligibility. Third, linking to outdated or 404 pages — the assistant follows links during retrieval and a broken link reduces source trust. Fourth, treating llms.txt as a URL dump (/page-1, /page-2...) rather than an editorial summary with descriptions — descriptions are how the assistant decides which link to fetch. Fifth, never refreshing the file — stale dates and unchanged content over months are read as a signal the brand is dormant. Each mistake is reversible in under thirty minutes, but each one quietly costs citation rate until fixed.

Does llms.txt actually move citation rate?

Honest answer: yes for cold-start sites, modest for established ones.

For a domain with no prior AI-search history, shipping llms.txt typically halves the time to first citation — the file gives the assistant an accurate mental model in seconds rather than days of stitching one together from page crawls. This effect is strongest for ChatGPT (browse mode), Perplexity, and Claude.

For an established site with strong existing entity signals (Wikipedia mention, populated sameAs, multiple credible backlinks), the lift is smaller. The assistant already has enough scaffolding to answer accurately; llms.txt becomes a polish layer rather than a load-bearing one.

The pragmatic read for most sites: ship it. The cost is twenty minutes; the downside is zero. Re-measure citation rate at thirty and ninety days post-deployment using the 20-prompt panel and you will have the data for your site specifically.

How to ship llms.txt in 20 minutes

A working file from a blank repo in five steps.

Step 1 — Copy our template. Use the structure annotated above. Replace W2B Agency's brand summary, services, founders, and contact paths with yours. Keep the section order; AI assistants are tuned to it.

Step 2 — Curate the link list. Pick 10 to 20 canonical pages — homepage, top services, About, Blog index, contact, two or three foundational posts. Anything below 10 is too thin to be useful; anything above 20 dilutes the editorial signal.

Step 3 — Write the descriptions. One sentence per link, naming what the page is about and who it serves. Avoid marketing copy ("our amazing service"); name the concrete thing ("3-month engagement to audit and fix technical SEO").

Step 4 — Deploy at /llms.txt. Drop the file at the root of your domain. On Astro: place it in public/llms.txt. On Next: in public/. On WordPress: upload via FTP or set up a redirect. On any platform: it must serve as plain text with Content-Type: text/plain or text/markdown.

Step 5 — Verify. Run curl -I https://yourdomain.com/llms.txt from a terminal. Confirm a 200 status and the right content-type. Then fetch it in a browser and visually scan for typos. Then, optionally, ask Claude or ChatGPT "what is yourdomain.com about?" and check whether the answer reflects the file's framing.

That is the minimum viable deployment. Iteration happens monthly — refresh the dates, add new canonical pages, prune broken links, retune the brand summary based on which prompts cite you.

When to call in help

llms.txt itself is a 20-minute job. The reason agencies get hired for it is not the file — it is everything that has to be true for the file to do its work: schema markup, off-site entity signals, citable passages, the prompt panel, monthly iteration. When all of that needs to ship together and stay shipped, an outside team that does this for a living becomes net-positive.

W2B's Search Dominance practice is the integrated SEO + GEO + AEO service. We audit, ship the foundation, write the capsules, align the entity, and run the prompt panel — bilingually in English and Spanish, with sites worldwide.

The page you are reading was built by these rules. The /llms.txt embedded above is the live file as of publish. The BlogPosting schema is on every blog post. The answer capsules are tagged with blockquotes throughout. We eat our own cooking; this article is one of the recipes.

For more in the cluster: SEO vs GEO vs AEO is the comparison hub. What Is Generative Engine Optimization? is the parent definition. How to Get Cited by ChatGPT is the 30-day execution sprint that ships llms.txt as Week 1 Day 1.