• llms.txt is a Markdown file at your domain root that points AI systems to your best content. It is not access control, and as of June 2026 Google confirms it plays no role in Search, AI Overviews, or AI Mode.
  • The measurable payoff for AI citations is close to zero today, but adoption is rising and agent tooling already reads it, so it is a cheap, forward-looking bet rather than a ranking lever.
  • Real AI visibility comes from content good enough to be cited, clean technical foundations, and genuine authority. llms.txt is one small layer on top of that, not a shortcut.

A year ago, llms.txt looked like the next robots.txt. Marketers rushed to ship one, half of them describing it as a way to control how AI uses their content. Then the people who actually run the crawlers started talking, and the story got more complicated. Google said skip it. The citation data came back almost empty. And yet the file keeps spreading, plugins now generate it by default, and Google’s own audit tool checks whether you have one. So which is it: dead idea or quiet infrastructure for the agent web? The honest answer sits in between, and it matters for how you spend your time.

What is llms.txt, really?

llms.txt is a plain Markdown file you place at the root of your domain, at yourdomain.com/llms.txt. Inside it, you give AI systems a short, curated map of your most important pages, usually with a one-line brand summary at the top and grouped links below. Jeremy Howard of Answer.AI proposed it in September 2024. The idea was simple. Instead of forcing a model to crawl, scrape, and guess its way through your entire site, you hand it a clean index of what matters.

The single most common mistake is treating it like robots.txt. It is not. robots.txt tells bots what they may and may not access, and it is the only file search engines officially recognize for that job. llms.txt does the opposite kind of work. It does not block, gate, or set permissions. It recommends. If you want to control AI crawler access, that still lives in robots.txt and your AI bot user-agent rules. Keep the two jobs separate, because conflating them is where most bad llms.txt advice begins.

Why did everyone get excited, and why did Google say no?

The excitement made sense. Search shifted hard toward AI answers, and site owners wanted a lever they could pull to stay visible. A single file you write in ten minutes felt like that lever. Adoption climbed fast. By spring 2026, Google was indexing somewhere between 120,000 and 200,000 llms.txt files, and studies put adoption at roughly 5 to 15 percent of sites, led by SaaS, developer tools, and content publishers.

Then Google’s Search team drew a hard line. In its May 15, 2026 AI optimization guide, Google explicitly listed llms.txt among the tactics site owners can ignore. The reasoning is structural: AI Overviews and AI Mode pull from the same Google Search index that classic ranking uses, so a separate file changes nothing about what those features see. Gary Illyes confirmed at Search Central Live that Google does not support llms.txt and has no plans to. John Mueller went further, comparing it to the old keywords meta tag, the one search engines abandoned years ago because site owners controlled it and therefore gamed it.

The independent data backs Google up. An ALLMO analysis of nearly 95,000 cited URLs across 11,867 AI responses found exactly one citation that pointed to an llms.txt page. Server-log studies tracking the bots that actually drive citations, GPTBot, ClaudeBot, PerplexityBot, found requests to llms.txt to be statistically negligible. If the promise was higher citation rates, the evidence does not support it yet.

So is llms.txt dead?

No, and this is where the simple “Google said no, ignore it” take gets lazy. Two things are true at once. The file does nothing for your Google AI visibility today, and it is becoming standard plumbing for a different layer of the web.

That layer is agentic browsing. The AI coding assistants and agents people now use every day, Cursor, Claude Code, Copilot, fetch llms.txt to pull the right documentation with less wasted compute. Google itself signaled this split. Its Lighthouse tool, version 13.3 released in May 2026, moved an Agentic Browsing category into its default audit, and that audit checks whether your site serves a valid llms.txt. So Google’s Search team tells you to skip it while Google’s developer tooling checks that you have it. Both positions are coherent once you separate search ranking from agent access. They are not the same game.

This is the practical read for June 2026. llms.txt is a low-cost, low-current-yield bet with clear optionality. Writing one takes under an hour. It will not lift your rankings or your citation rate right now. But if a major provider decides to respect it, and the robots.txt precedent shows conventions sometimes do get adopted after the fact, you are already correct. And even if that day never comes, you are left with a clean, machine-readable summary of your brand that any agent can read when asked. That has quiet value on its own.

How do you implement llms.txt properly?

The format is the part people overcomplicate. It is Markdown, not directives. Start with an H1 carrying your brand name, add a one-line summary as a blockquote, then group your highest-value links under H2 sections with a short description after each. Keep it curated. The whole point is to point AI at your best 20 to 50 pages, not to mirror your sitemap. Dumping every URL defeats the purpose, which is signal, not volume.

Here is the basic shape:

# Content Managers

> AI-powered content, automation, and digital marketing for growing businesses.

## Core guides
- [LLM Optimization Guide](https://www.content-managers.com/insights/how-to-optimize-your-website-to-be-llm-friendly/): How to make a site AI-friendly
- [How AI Agents Are Transforming Content Marketing](https://www.content-managers.com/insights/how-ai-agents-are-transforming-content-marketing/): What agents change for content teams

## Services
- [AI Search Visibility](https://www.content-managers.com/digital-marketing-services/): Getting cited by AI assistants

A few rules keep it clean. Write descriptions for context, not keywords, so “how we approach AI visibility” beats a stuffed phrase. Skip gated pages, since AI cannot read behind a login anyway. Update it quarterly or whenever you restructure, because dead links to deleted pages read as neglect. One more tip from John Mueller: add a noindex to the file itself so it does not show up as a stray page in search results while still staying available to any agent that requests it. If you run WordPress, plugins like Yoast now generate the file automatically, which removes the manual upkeep entirely.

Where should llms.txt sit in your actual strategy?

At the bottom of the priority list, and that is not an insult. It is a cheap finishing touch on a house that needs to be built first. The lever for AI visibility was never a file. It is whether AI systems judge your content good enough to cite, and nothing in a text file changes that judgment.

The work that does move the needle is the same work that has always mattered, now aimed at a new reader. Write original, experience-led content that answers real questions completely, because AI Mode fans a single query into many sub-questions and rewards pages that cover the whole shape of a topic. Name your entities and cite your sources, so a model can verify and trust you. Keep your technical foundations clean so crawlers reach everything. If you want the fuller picture of how this fits together, our guide on optimizing your website to be LLM-friendly covers the content and technical layers, and our breakdown of how AI agents are transforming content marketing explains why agent-readable infrastructure is becoming its own discipline.

The honest framing for any client asking about llms.txt: ship it because it is cheap and forward-compatible, then spend your real hours on content quality, closing the content gaps your audience searches for, and measuring whether AI actually mentions you. If you want help building that full visibility layer rather than just the file, that is exactly what our AI search visibility work is built for.

How do you measure whether any of this works?

Stop guessing and measure citations directly. The question that matters is whether AI assistants name your domain when someone asks about your category, and you can test that. Ask the same buying-intent questions your customers ask across ChatGPT, Perplexity, and Gemini, and record whether you appear. That manual check is low-tech and still the truest reflection of the real user experience.

Beyond manual testing, a category of AI visibility tools now tracks citation share across AI surfaces and flags which prompts trigger competitors instead of you. Watch your server logs too, filtering for the AI user-agents that drive citations, to see who is actually fetching your content. The point is a measurable baseline instead of hope. Once you can see where you stand, you know whether your content is earning its place in AI answers or whether you are shipping files into the void. For the wider toolkit, our roundup of the best AI tools for content marketing and our walkthrough of competitor analysis both feed into a real visibility picture.

The takeaway

llms.txt is not the robots.txt of AI, and it is not the GEO shortcut it was sold as. It is a small, cheap, forward-looking file that does little today and might do more tomorrow. Ship it in your lunch break, noindex it, keep it curated, and then forget about it. Because the real contest is not whether you have the file. It is whether your content is the answer when someone asks an AI about your category. Win that, and the file is a footnote. Lose it, and no file will save you.