llms.txt and AI crawler control: what actually works in 2026
I wired llms.txt, an AI-crawler allow-list, and IndexNow into my own site. Here is what each one really does, what answer engines quietly ignore, and what is worth your time.
On this page
Three files claim to control how AI sees your site: robots.txt, llms.txt, and an IndexNow key. Only two of them actually do anything for answer engines today. I wired all three into this site, so here is the honest version of what each one is for and what to bother with.
robots.txt: the file AI crawlers actually obey
The major AI crawlers respect robots.txt. OpenAI's GPTBot, Anthropic's ClaudeBot, PerplexityBot, and Google-Extended all read and honour it, so this is your real control surface. There are two strategies: block the training bots to keep your content out of model training, or allow the retrieval bots so your pages can be pulled into live AI answers.
For GEO the choice is easy: you want the answer engines to see and cite you, so allow the retrieval and search agents. One catch worth knowing: Google-Extended has no separate crawler you can block at the server, so robots.txt is the only lever that works for it (see Google's crawler docs). On this site I allow all of them, because being cited and being trained on both help visibility.
llms.txt: what it promises versus what happens
llms.txt is a proposed convention (from llmstxt.org, introduced by Jeremy Howard of Answer.AI in September 2024): a Markdown file at /llms.txt that hands an LLM a clean, curated map of your site instead of making it wade through HTML.
The honest 2026 reality: it is not yet a ranking or citation lever. Adoption sits around 10% of sites in one study of 300,000 domains, and as of this year no major AI provider (OpenAI, Google, Anthropic) has committed to reading it in production; the crawlers mostly skip the file and parse your HTML anyway (Search Engine Land). Where it genuinely gets used is coding agents: Cursor, Claude Code, and Copilot fetch llms.txt when pointed at docs.
So why did I add one? Because it costs almost nothing, it stays in sync when generated from your content, and it signals intent. It is cheap future-proofing, not a growth tactic. Do not expect it to move citations this year.
IndexNow: speed to Bing, and therefore ChatGPT
IndexNow lets you ping search engines the moment a URL changes instead of waiting for a crawl. It matters for GEO because ChatGPT Search and Copilot run on Bing's index, so getting into Bing quickly is a direct answer-engine input. I wired it to fire on every deploy, which is a genuine, low-effort win.
The three files at a glance
| File | What it is for | Respected by AI engines in 2026? | Worth doing? |
|---|---|---|---|
robots.txt (AI rules) | allow or deny specific AI bots | Yes, honoured by the major bots | Yes, essential |
llms.txt | a curated content map for LLMs | Rarely (coding agents, yes) | Optional, cheap future-proofing |
| IndexNow key | instant URL submission to Bing | Yes, via Bing/IndexNow | Yes, easy win |
Where to start
Do the two that work today: set your robots.txt to welcome the retrieval bots, and wire up IndexNow. Add llms.txt only because it is cheap, not because it will get you cited. The thing that actually earns citations is still what it has always been: clear, well-sourced content that other sites talk about, which is the real point of GEO versus AEO versus SEO and getting cited in AI answers.
Want this set up and audited on your own site? Here is how I run technical GEO audits.
Further reading
Keep reading
GEO vs AEO vs SEO: the 2026 definitions
SEO, AEO, and GEO are three overlapping jobs, not synonyms. Here is what each one actually optimises for, with a side-by-side comparison and where to start.
GEO and AEO: How to Get Your Brand Cited in AI Answers
Search is splitting in two. Here is the practical playbook I use to get brands cited inside ChatGPT, Perplexity, and Google AI Overviews, not just ranked on page one.