llms.txt is a markdown file you drop in the root of your website. That’s it. One file, plain text, sitting at yourdomain.com/llms.txt.
The idea: give AI systems a clean, structured guide to your site — which pages matter, how to describe your project, what to skip. Think of it as a curated index for language models instead of making them crawl thousands of pages on their own.
Jeremy Howard from Answer.AI proposed it in September 2024. The SEO community immediately called it “the new robots.txt for AI.” Agencies started selling it as a GEO must-have. Crypto projects added it en masse. And three large-scale studies later, we have actual data on what it does.
What’s Actually Inside the File
A standard llms.txt contains a brief description of the project, links to the most important pages, and optional context about how the site is organized. For a crypto protocol, that might look like: a one-paragraph summary of what the protocol does, links to the whitepaper, docs, governance page, and audit reports, plus a note distinguishing the mainnet from testnet documentation.
# Protocol Name
> One-sentence description of what the protocol does and its core value proposition.
## Docs
- [Whitepaper](https://protocol.xyz/whitepaper): Full technical specification
- [Documentation](https://docs.protocol.xyz): Developer docs and API reference
- [Audit Reports](https://protocol.xyz/audits): Security audit history
## Governance
- [Governance Portal](https://gov.protocol.xyz): On-chain proposals and voting
- [DAO Forum](https://forum.protocol.xyz): Community discussion and signaling
## Optional
- [Testnet Docs](https://docs.protocol.xyz/testnet): Testnet-specific setup
- [Changelog](https://protocol.xyz/changelog): Protocol version historyThere’s also llms-full.txt — a heavier version that includes the full content of key pages rather than just links. Same idea, more data. Neither has confirmed platform support in production.
llms.txt vs robots.txt vs sitemap.xml
Three different files, three different purposes.
robots.txt tells crawlers where they can and can’t go. It’s a protocol enforced by convention — Google, Bing, and most crawlers actually follow it. It affects which pages get indexed.
sitemap.xml tells search engines which URLs exist on your site and when they were last updated. It’s a discovery aid. Search engines may or may not follow it.
llms.txt has no enforcement mechanism at all. No crawler is required to read it. No platform has committed to following it. It’s a suggestion with no recipient that’s confirmed to be listening.
For crypto projects: robots.txt and sitemap.xml are non-negotiable technical foundations. llms.txt is optional and, based on current data, has no confirmed effect on how AI systems behave.
How Adoption Looks in 2026
The file spread fast. SE Ranking found it on 10% of 300,000 analyzed domains. Ahrefs saw it on 28% of 137,000 sites — the higher number reflects a more tech-forward sample.
That’s a meaningful adoption curve for something under two years old. But adoption driven by speculation is different from adoption driven by confirmed utility. Most teams added the file expecting platforms to start using it — not because platforms had announced they would.
Do AI Platforms Actually Read It?
This is the question that matters. Here’s where each major platform stands as of late 2026.
Google officially said in May 2026 that llms.txt “has no effect on ranking or visibility” and that creating it is “something that can be ignored.” John Mueller added in June that it “won’t help LLMs differentiate sites.”
OpenAI hasn’t documented it as a citation signal anywhere. Their public guidance focuses on allowing OAI-SearchBot in robots.txt — that’s what actually helps their crawler reach your content.
Anthropic mentioned it in documentation but without any claim about citation effects.
Perplexity has partial support — the file can influence which pages Perplexity prioritizes when crawling. That’s page selection, not citation frequency.
So: no major platform has confirmed it as a production signal for how often they cite your brand.
What the Server Logs Show
Ahrefs ran the most direct test: actual server log analysis across 137,000 sites, looking at what visited llms.txt files in May 2026.
97% of published files received zero requests that month. Of the 3% that got any traffic at all, 96% of those requests came from generic bots — not AI crawlers. ChatGPT and Perplexity bots together made up 1% of all requests to llms.txt files. Another 12% were GEO checker tools and researchers specifically looking for the file.
Ahrefs’ conclusion: “If you publish llms.txt today, the most likely outcome is that nothing will ever request it.”
And critically — AI bots didn’t go looking for files that didn’t exist. A 404 at /llms.txt attracted zero AI traffic. So there’s no evidence of active crawling for it either.
What the Citation Studies Show
Three research teams analyzed whether having llms.txt correlates with appearing in AI-generated responses.
SE Ranking — 300,000 domains. No statistically significant correlation between llms.txt presence and AI citation frequency. When they built a predictive model for which domains get cited by AI and included llms.txt as a variable, removing it improved the model’s accuracy. The variable added noise, not signal.
ALLMO — 94,614 cited URLs across five AI platforms. One of those URLs was an llms.txt page. Zero of the top 50 most-cited media domains globally use llms.txt. Zero of the top 20 brands appearing in German ChatGPT responses use it. One domain in the top 50 globally (Target.com) has the file.
The combined picture: 574,000+ domains analyzed, three methodologies, one finding. Having llms.txt doesn’t predict AI citation frequency.
How LLMs Actually Learn About Your Protocol
This is the part most guides skip, and it explains why llms.txt doesn’t work the way people expect.
Language models don’t discover your protocol by crawling your site at query time. They learned about everything they know during pretraining — a process that happened months or years before anyone asked them a question. That training data is a snapshot of the internet: articles, documentation, forum posts, GitHub repos, academic papers, YouTube transcripts.
When someone asks ChatGPT about your Layer 2, the model retrieves what it absorbed during training. It doesn’t go fetch your llms.txt. For retrieval-augmented systems like Perplexity, real-time web results do factor in — but those come from traditional search indexing infrastructure, not from reading llms.txt files.
So if you want AI to know about your protocol, the actual mechanism is: be mentioned in places that make it into training data and search indexes. Reddit, Wikipedia, media coverage, GitHub, YouTube. That’s the pipeline. A file sitting on your server that nothing reads isn’t part of it.
So Does llms.txt Affect AI Citations?
Based on current data: No ❌
Not because the concept is wrong — the idea of giving AI systems a structured site guide is reasonable. But because the infrastructure to use it isn’t in place at any mainstream platform in production. The file exists. The readers don’t.
The Myths That Spread Anyway
The file has zero confirmed effect on citations. But the claims around it haven’t slowed down. Here’s what’s wrong with each of them.
“Without llms.txt, LLMs ignore your protocol.” LLMs don’t discover protocols at query time. They learned during pretraining — from Reddit, GitHub, media, YouTube, documentation that got indexed. If your protocol isn’t in that corpus, a file on your server won’t fix it. If it is, the file adds nothing. Either way, llms.txt isn’t the variable.
“Add llms.txt and you’ll get more AI citations.” SE Ranking tested this directly on 300,000 domains. No correlation. And when they built a machine learning model to predict which domains get cited by AI, including llms.txt as a variable made the model less accurate. Removing it improved predictions. The file didn’t just fail to help — it added noise.
“Top crypto projects already use it. You’re behind.” ALLMO analyzed 94,614 URLs cited across five AI platforms. One was an llms.txt page. Zero of the top 50 most-cited media domains globally use it. Zero of the top 20 brands surfacing in German ChatGPT responses use it. The projects getting cited in Perplexity and ChatGPT aren’t succeeding because of the file. They’re succeeding despite not having it.
“llms.txt controls how AI describes your protocol.” The claim is that if you specify your terminology and positioning in the file, AI systems will adopt it. But LLMs draw from training data when generating responses — not from files they’re not reading. For retrieval-augmented systems like Perplexity, real-time results come from search indexing infrastructure, not llms.txt. The only way to influence how AI describes your protocol is to shape what third parties write about it.
“It’s low-effort, no-risk — why not add it?” Technically true. The file doesn’t hurt anything. But it creates a false sense of progress on AI visibility. Teams implement it, check the box, and move on — believing GEO is handled. Meanwhile the factors that actually drive citations stay unaddressed. For early-stage protocols where visibility decisions compound, that’s not a neutral outcome. Building the signals that actually register is what AI marketing for crypto is built around.
What Actually Drives LLM Citations for Crypto Projects
The same research that tracked llms.txt also measured what does correlate with AI citation frequency.
Brand mentions across third-party domains: 0.664 correlation. The more your protocol gets mentioned on external sites — not by you, but about you — the more AI systems cite it. Ahrefs measured this across 75,000 brands: the gap between high-mention and low-mention brands in AI citation frequency is not marginal, it’s an order of magnitude. For crypto: editorial mentions in CoinDesk, The Block, or Decrypt, threads on Reddit where your protocol comes up organically, posts on X that make it into indexed content. All of it feeds the signal.
YouTube: 0.737 correlation — the strongest single factor in Ahrefs’ dataset. AI systems read transcripts and descriptions. A protocol explainer, a founder interview, a technical walkthrough. These aren’t just marketing — they’re training data.
Wikipedia. Brands with a Wikipedia article get cited by AI far more often than those without one. The gap is large enough that for an established protocol with real on-chain history and community, a well-sourced Wikipedia page is probably the highest-ROI move available for AI visibility.
Reddit: 3.9× multiplier. SE Ranking analyzed 129,000 domains. Brands with millions of Reddit mentions average 7 ChatGPT citations. Brands with minimal Reddit presence average 1.8. Reddit feeds training corpora directly — it’s one of the most indexed, most cited sources in LLM training data.
Brand search volume. When people search for your protocol by name, it signals real-world traction. Brand search volume tracks closely with AI citation frequency — it’s a leading indicator, not just an SEO metric.
Backlinks are the weakest off-site signal in the dataset. Not irrelevant, but not what moves AI citations. AI systems don’t evaluate content by link equity the way Google does. They work from accumulated mentions and cross-source validation.
Earned media vs. owned content. The overwhelming majority of AI brand mentions happen through third-party pages, not the brand’s own site. For a crypto protocol, what others write about you outweighs what you write about yourself — by a wide margin.
For comparison: llms.txt came in at approximately 0. It’s not slightly weaker than the other signals. It doesn’t register.
Content Structure That Helps AI Extract Information
Beyond external presence, how you write your own content affects how AI systems pull from it.
Self-contained chunks of 50–150 words with specific claims get extracted by AI far more reliably than long, unstructured text. AI systems pull passages, not pages — a 2,000-word documentation page written as one block is much harder to cite than the same information broken into discrete, answerable sections.
FAQPage schema feeds AI question-answer extraction directly. If someone asks “how does [Protocol] handle MEV?” and you have a FAQ entry that answers that in one clean paragraph, it’s extractable. If the answer is buried in a 4,000-word technical overview, it probably won’t surface.
Content with concrete data — specific numbers, audit results, dated TVL history — gets extracted by AI more often than generic explanations. Comparison tables with clean HTML markup and pull quotes tied to specific claims are particularly extractable formats. For a DeFi protocol, these aren’t just UX decisions. They’re the structural choices that determine whether AI can cite you at all.
The Ghost Citation Problem
Semrush analyzed 3,981 domains and found that 62% of AI citations are “ghost citations” — the source gets used, the brand doesn’t get named. AI cites the information but attributes it generically or not at all.
For crypto projects building recognition, this matters. The fix is content where your protocol name is grammatically inseparable from the claim. “The [Protocol] approach to MEV protection” is harder to cite without attribution than “MEV protection methods.” Write so that stripping the brand name breaks the sentence.
When llms.txt Does Make Sense
Three cases where publishing it is reasonable.
Developer-facing products. If your primary audience is developers integrating your protocol, adding llms.txt follows the convention that audience expects. Some coding assistants show clearer support for it.
Long-term optionality. Common Crawl added llms.txt to its crawler’s seed list in July 2026. Models trained on future Common Crawl snapshots may see your file. Publishing now is a cheap hedge against that becoming meaningful. It’s not a 2026 growth tactic — it’s a small bet on 2027 or later.
Closing the technical checklist. If your team is already doing the things that drive real AI citations — earned media, Reddit presence, structured content, schema markup — adding llms.txt takes a few hours and rounds out the GEO setup. The issue isn’t that it costs too much. It’s when it replaces the other work, not supplements it.
At ICODA we run AI citation audits across ChatGPT, Perplexity, and Gemini for crypto clients. llms.txt almost never shows up as a gap. What shows up: low third-party mention volume, missing Wikipedia presence, unstructured documentation, and no FAQPage schema on high-intent pages. Those are the actual gaps.
Frequently Asked Questions (FAQ)
A markdown file placed at the root of a website to help AI systems understand the site’s structure and content priorities.
No, based on current data. SE Ranking found zero statistically significant correlation across 300,000 domains. ALLMO found that none of the top 50 most-cited global domains use it (except one). Ahrefs confirmed that 97% of files receive zero AI crawler traffic.
Perplexity has partial support for page prioritization during crawling. ChatGPT has not documented it as a signal. Both bots together accounted for 1% of all requests to llms.txt files in Ahrefs’ May 2026 server log analysis.
Third-party brand mentions, YouTube presence, Wikipedia, Reddit, and brand search volume — in that order of measured impact, per Ahrefs and SE Ranking research. On-site: self-contained content chunks, FAQPage schema, data-backed claims, and comparison tables.
If you have a few hours and you’re already doing the work that drives actual AI visibility — yes, add it as a hedge. If you’re choosing between llms.txt and building brand presence on third-party platforms, do the brand presence work first.
No. GEO is the full discipline of optimizing for AI citation and visibility. llms.txt is one small technical element within it — currently with no measured effect in production systems.
Share with
Rate the article
