llms.txt and AI crawler control: what actually works in 2026

Three files claim to control how AI sees your site: robots.txt, llms.txt, and an IndexNow key. Only two of them actually do anything for answer engines today. I wired all three into this site, so here is the honest version of what each one is for and what to bother with.

robots.txt: the file AI crawlers actually obey

The major AI crawlers respect robots.txt. OpenAI's GPTBot, Anthropic's ClaudeBot, PerplexityBot, and Google-Extended all read and honour it, so this is your real control surface. There are two strategies: block the training bots to keep your content out of model training, or allow the retrieval bots so your pages can be pulled into live AI answers.

For GEO the choice is easy: you want the answer engines to see and cite you, so allow the retrieval and search agents. One catch worth knowing: Google-Extended has no separate crawler you can block at the server, so robots.txt is the only lever that works for it (see Google's crawler docs). On this site I allow all of them, because being cited and being trained on both help visibility.

llms.txt: what it promises versus what happens

llms.txt is a proposed convention (from llmstxt.org, introduced by Jeremy Howard of Answer.AI in September 2024): a Markdown file at /llms.txt that hands an LLM a clean, curated map of your site instead of making it wade through HTML.

The honest 2026 reality: it is not yet a ranking or citation lever. Adoption sits around 10% of sites in one study of 300,000 domains, and no major AI provider (OpenAI, Google, Anthropic) reads it in production; Google has explicitly said it will not, comparing the idea to the long-discredited keywords meta tag, and the crawlers mostly skip the file and parse your HTML anyway (Search Engine Land). Where it genuinely gets used is coding agents: Cursor, Claude Code, and Copilot fetch llms.txt when pointed at docs.

So why did I add one? Because it costs almost nothing, it stays in sync when generated from your content, and it signals intent. It is cheap future-proofing, not a growth tactic. Do not expect it to move citations this year.

IndexNow: speed to Bing, and therefore ChatGPT

IndexNow lets you ping search engines the moment a URL changes instead of waiting for a crawl. It matters for GEO because ChatGPT Search and Copilot run on Bing's index, so getting into Bing quickly is a direct answer-engine input. I wired it to fire on every deploy, which is a genuine, low-effort win.

The three files at a glance

Where to start

Do the two that work today: set robots.txt to welcome the retrieval bots, and wire up IndexNow. Ship llms.txt too, but treat it as a footnote, not a strategy, and put the real effort into the content itself, which is what actually gets quoted in AI answers. These three files are just the access layer; they sit inside the wider setup I walk through in technical SEO in 2026.

If you want your crawler access audited alongside the rest of your technical GEO setup, that is exactly what I do for clients.

Further reading

• The llms.txt proposal (llmstxt.org)
• Google Search Central: overview of Google crawlers
• Search Engine Land: llms.txt, a proposed standard