llms.txt is a plain Markdown file at yoursite.com/llms.txt that gives AI tools a short, curated list of your most useful pages. Google says Google Search ignores it, and no major AI search engine has said it uses one to choose sources. It clearly helps documentation and developer sites that coding assistants read; for most small businesses it's an optional hour of work, not a priority.
Someone at a conference, a plugin notice or a sales email has told you your site “needs an llms.txt” to show up in ChatGPT. It sounds like robots.txt for AI, it sounds urgent, and nobody can quite say what it does. So you’re left wondering whether you’re missing something or being sold something.
Here’s the straight answer we give clients: what the file is, who actually reads it as of November 2026, where it earns its keep and where it doesn’t, and what to do if you decide to add one.
What is llms.txt?
llms.txt is a proposal, not a standard. Jeremy Howard, co-founder of Answer.AI, published it on September 3, 2024 at llmstxt.org. The idea is simple: web pages are built for people, with menus, scripts, cookie banners and ads. A language model working with a limited amount of text struggles to find the useful part. So a site can publish one small Markdown file that says who it is and links to the pages worth reading, ideally in clean Markdown versions.
The proposal is aimed at the moment an AI tool is answering a question (what it calls inference), not at training. Howard updated it to a second version in August 2026, after two years of real use. The changes say a lot about who uses it: the new text describes agents reading the file and following its links, says coding agents use these files reliably, and drops the mechanical meaning of the “Optional” section (llmstxt.org changes).
What goes in the file
The format is ordinary Markdown in a fixed order. Only the first line is required.
- An H1 with your name. The one required part.
- A blockquote summary. One or two sentences with the context a reader needs for everything below.
- Optional notes in plain text. Short facts that help a tool answer correctly. No headings here.
- H2 sections of links. Each line is a Markdown link, then a colon and a few words on what the page covers.
- An “Optional” section. Secondary links a tool can skip when it’s short on space. In version 2 this is a convention, not a rule.
The proposal also asks sites to offer a Markdown copy of important pages at the same address with .md added. You’ll also see llms-full.txt, a single file with the full text of every page. That one isn’t part of the proposal. Documentation platforms made it popular, and it can get huge: Anthropic’s developer docs publish an llms-full.txt of about 49 MB, far more than a person or a small business would ever need.
What llms.txt is not
It isn’t robots.txt. It doesn’t allow or block anything. Access for AI crawlers is still controlled in robots.txt, and that’s what OpenAI, Anthropic and Perplexity document for their bots (OpenAI, Anthropic, Perplexity). If your robots.txt blocks OAI-SearchBot, a perfect llms.txt won’t get you into ChatGPT search.
It isn’t a sitemap. A sitemap lists every page for search engines. llms.txt is a short, hand-picked list for language models.
It isn’t a ranking factor. Not in Google, and there’s no published evidence that any AI assistant cites you more because you have one.
Don't treat llms.txt as a way to keep your content out of AI training or AI answers. It has no blocking power at all. If you want to limit a crawler, write the rule in robots.txt under that crawler's user agent, and check it with a robots.txt tester.
Who actually reads llms.txt today?
Google: Search ignores it, Chrome checks for it
Google’s guide to its generative AI features in Search lists “LLMS.txt files and other special markup” among the things you can skip. It says Google Search doesn’t use them, and that adding them “will neither harm nor help your site’s visibility or rankings in Google Search, as Google Search ignores them” (Google Search Central). That covers AI Overviews and AI Mode.
Google’s John Mueller made the same point more bluntly in 2025: “AFAIK none of the AI services have said they’re using LLMs.TXT,” and “To me, it’s comparable to the keywords meta tag” (Search Engine Journal).
The mixed signal comes from Chrome. In 2026, Lighthouse added an experimental Agentic Browsing category, and one of its audits checks for a summary file at the domain root (Chrome for Developers). That audit is about browser agents finding their way around a site. It sits outside the SEO score and says nothing about rankings.
OpenAI, Anthropic and Perplexity: they publish one, but don’t say they read yours
All three publish an llms.txt for their own developer docs, and so does Google’s Gemini API. None of their crawler documentation says their bots read a site’s llms.txt to decide what to fetch or cite. Their published controls are robots.txt rules per crawler (linked above). As of today, we couldn’t find an official statement from any of them that ChatGPT, Claude or Perplexity uses your llms.txt when answering questions.
The crawlers themselves
Server logs are the honest test, and the public data is thin. One published log census is useful as a snapshot:
Over 20 days in July and August 2026, dejan.ai logged 157 requests for its llms.txt file. Only 4 came from Google, Anthropic or OpenAI crawlers (3 from Googlebot, 1 from ClaudeBot, none from OpenAI), and just over half came from ordinary browsers (dejan.ai).
That’s one site, so don’t read too much into it. But it fits everything else on this page: the file gets the odd visit, and the pages themselves get the crawling.
Coding assistants: the clear win
The real readers are developers and their tools. A developer can paste a docs site’s llms.txt link into Claude Code, Cursor or a similar assistant, and the assistant pulls current, clean instructions instead of guessing from old training data. Stripe’s file even talks to these tools directly, telling them to check the npm registry for the latest package version “rather than relying on memorized version numbers” (Stripe llms.txt). That’s a use with an obvious payoff.
Does your site need an llms.txt?
It depends almost entirely on who reads your site and what they do with it.
| Type of site | Value of llms.txt | Why |
|---|---|---|
| Software docs, APIs, developer tools | High | Coding assistants are pointed at it and follow its links. |
| SaaS, e-commerce, firms with many services or policies | Low to moderate | Can help an agent find the right page. No proven effect on AI citations. |
| Local service business with 10 to 30 pages | Low | Your site is small enough to crawl as is. The hour is better spent elsewhere. |
For a plumber, a dental practice or a law firm, the pages decide whether AI assistants describe you correctly: a clear service page, a real service area, reviews, consistent details on your Google Business Profile, and robots.txt that lets the search crawlers in. Our guide to answer engine optimization covers that work in order.
What it costs to add one
In our experience, a first version for a small business takes about an hour: pick 10 to 20 pages, write a one-line note for each, upload the file to the root of your site. Some platforms will generate one for you; llmstxt.org lists Yoast SEO, AIOSEO and Wix among them. Generated files often dump every page and every post, so trim them to the pages that matter.
The real cost is upkeep. A file that lists a service you dropped, an old phone number or links that now redirect is worse than no file, because it’s exactly what an agent would trust. If you add one, put it on the same checklist as your sitemap and review it when pages change.
If you already have one, or want a draft to start from, check it before you publish:
It checks the file against the llmstxt.org format line by line, tests every link for errors and redirects, and checks that your robots.txt lets AI crawlers in. It can also build a first draft from your sitemap with each page’s real title and description. It can’t tell you whether any AI assistant will use the file, because nobody can measure that yet.
What we recommend
- If you run docs or a developer product: publish llms.txt and Markdown copies of your docs, generated by your build so they never drift.
- If you’re any other business and want one: keep it short and accurate. A name, a two-sentence summary, your key service pages, your about and contact pages. Full URLs only.
- Either way, put most of your effort into the pages. Answer the question at the top of each page, mark it up with schema that describes your business, and write for people first, as in our guide to writing that reads like a human. That’s what Google and the AI assistants actually crawl.
Don't pay for an "llms.txt optimization" package that promises AI rankings or citations. No AI company has said its answers use the file, so nobody can honestly promise a result from it.
Do this now
Curious how AI assistants talk about you right now? Our free AI Visibility Checker shows whether they mention your business and which sites they cite instead.
FAQs
Does Google use llms.txt?
No. Google’s guide to its AI features in Search says Google Search ignores llms.txt, so it neither helps nor hurts rankings or appearances in AI Overviews. Chrome’s Lighthouse does check whether the file exists, but that audit is for browser agents and isn’t part of SEO scoring.
Does ChatGPT read llms.txt?
OpenAI hasn’t said that ChatGPT or its crawlers use a site’s llms.txt to find or cite pages. Its crawler documentation describes robots.txt controls for OAI-SearchBot, GPTBot and ChatGPT-User. ChatGPT can read the file if a user gives it the link, as it would any page.
Is llms.txt the same as robots.txt?
No. robots.txt tells crawlers what they may and may not fetch, and OpenAI, Anthropic and Perplexity say their crawlers follow it (fetches a user asks for directly can be an exception). llms.txt only suggests which pages are worth reading, and it can’t block anything.
Where do I put an llms.txt file?
At the root of your domain, so it loads at yoursite.com/llms.txt as plain text. Version 2 of the proposal also allows files in subfolders, such as a docs section, where the most specific file applies to the pages under it.
What’s the difference between llms.txt and llms-full.txt?
llms.txt is a short index of links with notes. llms-full.txt, a convention popularized by documentation platforms rather than part of the proposal, puts the full text of the pages into one file. It’s useful for large docs sites and rarely worth it for a business site.
Sources
- The /llms.txt file, Jeremy Howard, llmstxt.org (v2, August 2026)
- Changes in llms.txt v2, llmstxt.org
- Google’s guide to optimizing for generative AI features on Google Search, Google Search Central
- Google says llms.txt comparable to keywords meta tag, Search Engine Journal, April 2025
- Lighthouse agentic browsing, Chrome for Developers
- Overview of OpenAI crawlers, OpenAI
- Does Anthropic crawl data from the web?, Anthropic
- Perplexity crawlers, Perplexity
- Crawler census: robots.txt and llms.txt, dejan.ai, August 2026
- Example files, checked November 3, 2026: Stripe llms.txt, Anthropic llms-full.txt, Gemini API llms.txt
