llms.txt is a proposed web standard that provides AI systems with a structured summary of a website’s most important content in a format optimised for large language model consumption. Placed at the root of a domain (yourdomain.com/llms.txt), it serves as a curated guide to a site’s key pages, written in Markdown rather than HTML, specifically designed to help AI systems understand and reference content efficiently. The proposal was created by Jeremy Howard of Answer.AI in 2024.
At Gorilla Marketing, we evaluate emerging AI search standards as part of our AI optimisation work for clients. The honest assessment of llms.txt right now is nuanced: it is easy to implement and there are early signals of platform interest, but current research shows no measurable impact on AI citations. This guide covers the full picture: what it does, what the evidence says and whether it deserves a place in your strategy.
How llms.txt Works
The llms.txt file sits at the root of a website, similar to robots.txt or sitemap.xml. But where robots.txt tells crawlers what to avoid and sitemaps tell them what exists, llms.txt tells AI systems what matters most and provides that content in a format optimised for LLM consumption.
The file uses a specific Markdown structure:
H1 heading: The site or project name
Blockquote: A brief description of what the site covers
H2 sections: Categories of content (e.g., “Documentation”, “Blog”, “Products”)
Links with descriptions: Under each section, links to key pages with brief descriptions of what each contains
Some implementations also include an llms-full.txt companion file that contains the full text of key pages in Markdown format. This gives AI systems direct access to the actual content without needing to crawl and render HTML pages. Evidence from Mintlify suggests that llms-full.txt is accessed more frequently by AI systems than the basic llms.txt file, likely because it provides the full content rather than just pointers.
How llms.txt Differs from robots.txt and Sitemaps
| File | Purpose | Audience | Format |
|---|---|---|---|
| robots.txt | Tells crawlers what not to crawl | Search engine bots | Plain text directives |
| sitemap.xml | Lists all pages for crawling | Search engine bots | XML |
| llms.txt | Curates key content for AI consumption | AI/LLM systems | Markdown |
robots.txt is about restriction. Sitemaps are about discovery. llms.txt is about curation, specifically selecting and presenting the content most useful for AI systems to understand and reference.
The key technical advantage is token efficiency. An HTML page uses thousands of tokens on navigation, scripts, styling and structural markup before the AI reaches the actual content. A Markdown equivalent conveys the same information in a fraction of the tokens, fitting more useful content within an LLM’s context window. For AI systems with limited context windows or token budgets per page, this efficiency matters.
Who Is Using llms.txt?
Adoption has grown rapidly but unevenly. Over 844,000 websites had implemented llms.txt as of October 2025. SE Ranking’s analysis of 300,000 domains found that approximately 10% had the file in place.
Notable early adopters include Anthropic (Claude’s parent company), Cloudflare, Vercel, Stripe, Hugging Face, Zapier and Cursor. The pattern is telling: adoption is concentrated among technology companies and developer-focused businesses. These organisations have content, particularly documentation, that is heavily consumed by AI systems and AI-powered coding tools.
Interestingly, high-traffic domains (100,000+ monthly visits) actually show lower adoption rates (8.27%) than mid-tier sites (10.54% for 1,001-5,000 visits). The assumption that industry leaders adopt first does not hold here. Mid-size tech companies have been the most active implementers.
CMS platforms are adding support. Yoast SEO, Wix and several static site generators now offer built-in llms.txt generation. As CMS integration spreads, adoption will likely accelerate regardless of proven impact, simply because the implementation barrier drops to zero.
What Does the Evidence Say About AI Citations?
This is the critical question, and the data is clear: there is no measurable citation impact yet.
SE Ranking’s 300,000 domain study found no correlation between llms.txt implementation and AI citation frequency. Their machine learning analysis using XGBoost went further: removing llms.txt variables from the model actually improved prediction accuracy. The file currently adds noise rather than signal for predicting AI citations.
ALLMO.ai’s analysis of 94,000+ cited URLs found no measurable citation uplift associated with llms.txt adoption.
OtterlyAI’s 90-day experiment monitored AI crawler behaviour on a site with llms.txt implemented. The result: only 84 AI bot visits to /llms.txt out of 62,100 total AI bot hits. That is 0.14% of AI crawler activity. AI bots visited the file extremely rarely compared to regular pages.
Google’s position is clear. Gary Illyes stated in July 2025 that Google does not support llms.txt and is not planning to. John Mueller separately compared it to the keywords meta tag, which was never used as a ranking signal. Google AI Overviews rely on traditional SEO signals, not llms.txt.
No major AI platform has publicly confirmed using llms.txt for citation decisions. OpenAI, Google and Anthropic have not stated that their systems read and use the file for source selection.
Where the Signals Are Mixed
The negative data above is robust, but there are counterpoints worth acknowledging.
Google included llms.txt in its A2A (Agent-to-Agent) protocol. This does not mean Google Search uses it, but it signals that Google sees llms.txt as relevant to the broader AI agent ecosystem. The A2A protocol is about how AI agents interact with websites, which is a different use case from search ranking.
Anthropic has specifically requested llms.txt implementation from partners. The company behind Claude sees value in the format for its AI systems, even if that value has not translated into measurable citation differences in studies.
Microsoft and OpenAI models actively crawl llms.txt files according to data from Profound. The bots visit the files even if their citation algorithms do not currently weight them.
Vercel reports that 10% of its signups now come from ChatGPT, partly attributed to broader AI optimisation efforts that included llms.txt. It is difficult to isolate llms.txt’s specific contribution, but the company considers it part of a working strategy.
Mintlify observes that documentation is now “50% for humans and 50% for LLMs”, and llms.txt is a natural extension of that reality for documentation-heavy sites.
The honest synthesis: llms.txt does not currently drive AI citations in any measurable way. But AI platforms are aware of it, some actively crawl it, and the broader trajectory of AI search suggests that structured, LLM-friendly content signals will matter more over time. The question may be when rather than whether, but the current evidence does not support treating llms.txt as a priority.
When Implementing llms.txt Makes Sense
Despite the lack of proven citation impact, there are situations where implementation is reasonable.
Developer documentation and technical content. If your site contains extensive documentation consumed by AI coding assistants (Cursor, Copilot, ChatGPT), providing structured Markdown helps these tools reference content more accurately. This is where the standard originated and where it has the clearest practical value today.
Large sites with complex content architectures. For sites with hundreds or thousands of pages, llms.txt serves as a curated directory pointing AI systems to the most important content. The curation exercise itself has strategic value, even independent of AI consumption.
Low-cost future-proofing. Creating a basic llms.txt file takes minutes. If AI platforms begin weighting the standard in the future, having it in place means you are ready. The effort is minimal and the downside risk is essentially zero.
As part of a broader AI strategy. llms.txt should not be the first or only action for AI visibility. But as one component alongside content quality, entity SEO, schema markup and answer-first content structure, it adds a small additional signal at negligible cost.
How to Create an llms.txt File
If you decide to implement, the process is straightforward.
Step 1: Identify key content. Select the 10 to 30 most important pages on your site. Focus on pages you want AI systems to reference: core service pages, key guides, documentation and high-value informational content. Prioritise pages that answer specific questions AI systems are likely to be asked about your business or industry. Do not include every page. The value of llms.txt is curation, not comprehensiveness. That is what your sitemap is for.
Step 2: Write the file. Use this structure:
“`
# Your Site Name
> A brief description of your site and what it offers.
Main Content
Page Title: Brief description of what this page covers
Another Page: Brief description
Resources
Guide Title: Brief description
“`
Step 3: Upload to root. Place the file at yourdomain.com/llms.txt. Ensure it is accessible via HTTPS and returns a 200 status code with content type text/plain.
Step 4: Optionally create llms-full.txt. If you want to provide full content access, create a companion file with the complete Markdown text of your key pages.
Step 5: Maintain it. Review and update the file quarterly or when significant new content is published. An outdated llms.txt is worse than none at all, because it points AI systems to stale content.
Common mistakes to avoid: do not list every page on the site (that defeats the purpose of curation), do not use HTML formatting (the file should be pure Markdown), do not forget the blockquote description (it provides essential context about the site’s purpose), and do not set it up once and forget it exists.
Where llms.txt Fits in an AI Strategy
llms.txt is a minor tactic, not a strategy. The activities that demonstrably improve AI citation rates, content quality, topical authority, original data, answer-first structure and entity clarity, should be the priority. Those fundamentals are covered in our guide to content formats for AI answers.
Think of llms.txt as the last 2% of an AI visibility strategy: worth doing if everything else is in place, but not where to start. And certainly not worth prioritising over content quality, structural optimisation or the other proven factors.
Gorilla Marketing’s AI optimisation services and technical SEO work include evaluation of emerging standards like llms.txt in the context of each client’s broader strategy. Get in touch if you want an honest assessment of where it fits for your business.
