The verdict: useful convention, unproven growth lever
The worst way to talk about llms.txt is to call it either magic or useless. It is neither. The fair verdict is narrower: llms.txt may be useful as a documentation and context convention, but it has no proven impact on AI citations today. That distinction matters because teams have limited time. If you only have half a day for AI search optimization, llms.txt is rarely the first job. You will usually get more value from checking your AI crawler robots.txt strategy, making priority pages available without login or rendering traps, adding clear answer blocks, citing source material, and building a repeatable prompt test set. A GEO task is worth prioritizing when three conditions line up: platforms acknowledge the mechanism, server logs or search data can observe it, and the work can change what users actually see in answers. llms.txt is weak on all three. It can exist on the site, but it should not become the headline win in a growth report.The evidence board
The public evidence points in the same direction: llms.txt has not been shown to increase AI citations. That does not mean it will never matter, and it does not mean every site should remove it. It means the file has not earned top-priority status.| Evidence | What was observed | Finding | Practical reading |
|---|---|---|---|
| SE Ranking study of 300,000 domains | Nearly 300,000 domains, llms.txt adoption, and AI citation frequency | 10.13% of domains had llms.txt, but statistical analysis and XGBoost feature importance did not find a meaningful citation lift | Strong macro signal: the file's presence is not a visible citation driver |
| WISLR 48-day server log analysis | 71,603 total requests, including 12,099 bot or AI crawler requests | No requests to /llms.txt or /llm.txt appeared during the sample | Single-site logs cannot speak for the whole web, but they show how the question can be verified |
| Google AI features documentation | Public guidance for AI Overviews and AI Mode | Google points site owners back to Search essentials and says no new machine-readable files or special AI markup are required | For Google AI features, llms.txt is not a published requirement |
| OpenAI crawler documentation and Anthropic crawler guidance | OAI-SearchBot, GPTBot, ChatGPT-User, ClaudeBot, Claude-User, and Claude-SearchBot behavior | Both companies document user agents, robots.txt handling, and request sources | The public control surface is crawling access, not a separate llms.txt citation switch |
| Search Engine Roundtable coverage of John Mueller's comments | Public comments from a Google Search representative | John Mueller said in June 2025 that no AI system was using llms.txt at that time and suggested checking logs | This is not a permanent policy statement, but it is a useful caution signal |
What llms.txt was built to do
The original llms.txt proposal was published by Jeremy Howard in September 2024. Its idea is simple: place a Markdown file at the site root so an LLM can find the important pages or documentation without wading through navigation, scripts, and noisy HTML. The proposal is especially relevant when a user asks an AI tool for help with a site, product, or developer resource. That is different from a ranking or citation signal. robots.txt tells crawlers which URLs they may request. A sitemap helps search engines discover URLs. llms.txt is closer to a curated context map. It says, “If an AI assistant needs a compact reading list, start here.” The SEO misunderstanding is understandable. The file sits in the root directory and looks technical, so people naturally compare it with robots.txt. But the purpose is different. Google's robots.txt documentation describes an access-control mechanism for crawlers. llms.txt describes a helpful context format. One controls access. The other organizes selected reading. If you run developer documentation, API references, open-source project docs, or complex product tutorials, llms.txt can still improve user experience. Someone might paste your llms.txt into Claude, ChatGPT, or Cursor and ask for implementation help. That is a real use case. It just is not proof of broad AI search citation growth.Why it fails as a GEO shortcut
llms.txt fails as a GEO shortcut for four reasons. First, major platforms have not publicly committed to using it as a citation, ranking, or answer-selection input. Second, many implementations are just URL lists that duplicate sitemap information. Third, AI search visibility depends on whether pages can be discovered, indexed, understood, and trusted. Fourth, the file can distract teams from the hard work: clearer pages, better evidence, and recurring measurement. Imagine a B2B SaaS site that adds a root-level llms.txt listing the homepage, pricing page, blog, and about page. That file does not answer the questions an AI system needs to answer: who the product serves, what problem it solves, how it differs from alternatives, what evidence supports the claims, and which pages should be cited for each claim. If the pages themselves are vague, a cleaner directory will not make them citable. GEO is not about giving the model a mysterious map. It is about making every candidate page clear enough to quote. The durable work is AI-search citable content: direct answers, definition blocks, comparison tables, sources, FAQs, video summaries, and stable retesting. llms.txt can point at those assets. It cannot replace them.When it is still worth shipping
You do not need to ban llms.txt. If your site architecture is already clean, adding a concise file is low risk. The trick is to keep it honest. Do not stuff it with keywords. Do not treat it as an authorization file. Do not report it as if it guarantees AI citations. It is worth shipping in three situations. First, your site has developer docs, API docs, open-source docs, or complex tutorials that users may hand to AI tools. Second, you have already handled the basics: crawlability, indexability, structured content, internal links, and source-backed pages. Third, you want a clean future-facing entry point in case more AI tools start supporting the convention. It should not come first when the site still has bigger problems: unstable sitemaps, blocked pages, login-gated documentation, conflicting product definitions, unsupported claims, or no fixed AI visibility test set. In that situation, llms.txt can create the comforting illusion that the team has “done AI SEO” while the pages themselves remain hard to cite.Where the same half-day should go
If you have three to four hours, spend them on work that platform documentation, logs, and page quality can all support.| Priority | Action | Why it beats llms.txt first | Success signal |
|---|---|---|---|
| P0 | Confirm priority pages are crawlable and indexable | AI answers still depend on accessible source pages | URLs return 200, are not noindexed, and are not blocked by robots or CDN rules |
| P0 | Add 40- to 90-word answer blocks to core pages | Clear answer units are easier to quote and summarize | Each main H2 can answer one question when copied alone |
| P1 | Add sources beside factual claims | Citation systems need verifiable material | Important statements point to docs, studies, cases, or a visible method |
| P1 | Build a fixed prompt test set | Without retesting, you cannot know whether visibility changed | The same prompts track mentions, citations, and errors month over month |
| P1 | Repair internal links and topic clusters | Search systems need relationships between pages | Related pages link naturally with descriptive anchors |
| P2 | Review AI bot server logs | Logs show whether crawlers are reaching the assets | You can see requests from OAI-SearchBot, Claude-SearchBot, PerplexityBot, or other relevant agents |
| P3 | Add a concise llms.txt | It preserves a clean context map for future or manual AI use | The file is accurate, short, and useful without keyword stuffing |
A 7-day replacement plan
Day 1: choose 10 important pages. Do not start with the whole site. Pick product pages, solution pages, tutorials, case studies, and category pages that can create business value. Day 2: write 30 prompts. Split them into brand, category, problem, competitor, buying, and implementation groups. Run the same prompts across the AI systems that matter to your audience. Day 3: rewrite page openings and H2 openings. Each page should answer one clear question near the top. Each major H2 should begin with a short, direct answer. Skip the generic opening filler. Day 4: add evidence. Use official documentation for platform behavior, public research for market claims, and real product pages or case material for product claims. Remove confident statements you cannot support. Day 5: add structure modules. Give each priority page at least one table, checklist, FAQ, or video summary. The goal is not decoration. The goal is faster evaluation by humans and AI systems. Day 6: check technical access. Review robots.txt, canonical tags, noindex tags, server status, CDN behavior, rendered HTML, and sitemap coverage. This is when adding llms.txt is reasonable, as a finishing pass. Day 7: rerun the same prompts. Track more than mentions. Record citations, cited URLs, answer accuracy, competitor appearances, and whether the next action points to a page you control. That is the data worth putting in a weekly GEO report.Decision scorecard: should you ship llms.txt this week?
Use this quick scorecard before assigning the work.| Question | Yes | No |
|---|---|---|
| Priority pages are crawlable, indexable, and not dependent on client-side text only | +2 | -3 |
| The site has a stable sitemap and clean internal links | +1 | -2 |
| Core pages include answer blocks, sources, FAQs, and citable tables | +2 | -3 |
| The site is documentation-heavy, API-heavy, or tutorial-heavy | +2 | 0 |
| The team already has an AI prompt baseline and monthly retest sheet | +2 | -2 |
| You plan to present llms.txt as a guaranteed AI citation lift | -4 | +1 |