Crypto projects get blocked from AI citation not because their content is weak or their domain authority is low — but because the same security infrastructure protecting them from scrapers is also refusing entry to every bot that would recommend them. You can have a polished whitepaper, active community, and solid tokenomics, and still be invisible in ChatGPT or Perplexity answers. Most teams don’t know it’s happening.
When we ran the AI Visibility Checker across 500+ crypto and Web3 sites, that’s the pattern we kept finding. Not a content problem. A configuration problem.
5 Numbers That Define Crypto’s AI Visibility Problem
- 68% of the 500+ crypto sites we audited block at least one major AI retrieval crawler — GPTBot, OAI-SearchBot, PerplexityBot, or ClaudeBot — most without realising it.
- 54% have zero JSON-LD schema across any key page type.
- Under 4% have an llms.txt file at their domain root.
- Average AI Visibility Score: 31 out of 100 — squarely in the “Poor” tier.
- Of blocked sites, 61% are blocked by infrastructure defaults — Cloudflare WAF rules, security plugins — not deliberate robots.txt policy.
The majority of sites in this dataset aren’t invisible because of weak writing. They’re invisible because a WAF rule applied during a server migration was never revisited. The fix isn’t months of link building — it’s a configuration change that takes an afternoon, if you know which layer to look at.
How the ICODA AI Visibility Score Works
The Checker scores each site across four categories:
AI Crawler Access (35%) — robots.txt directives for GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, Claude-SearchBot, Google-Extended, and ChatGPT-User, plus WAF-level blocking detection. WAF rules fire before robots.txt is read, so a permissive robots.txt with a blocking WAF rule achieves nothing.
Structured Data (25%) — JSON-LD schema presence, stale dateModified values, Microdata or RDFa flags. The checker scores this because it correlates with overall technical maturity — not because schema directly drives AI citations. More on that in Finding 2.
Technical Infrastructure (25%) — whether pages render as crawlable HTML without JavaScript execution; llms.txt presence; sitemap accessibility.
Content Accessibility (15%) — pages reachable without login walls, entity definition content present, content structure that allows AI systems to extract citable claims.
| Score | Tier | What it means |
|---|---|---|
| 0–44 | Poor | Blocked or near-invisible to AI retrieval systems |
| 45–64 | Fair | Inconsistent citations, mainly exact-match queries |
| 65–79 | Good | Regularly accessible; cited for relevant niche queries |
| 80+ | Excellent | Fully open; server-rendered; consistent AI citation |
Training Crawlers vs. Retrieval Bots
This distinction is where most fix attempts go wrong:
| Training Crawlers | Retrieval Bots | |
|---|---|---|
| Purpose | Collect content for model training | Fetch live content to cite in AI answers |
| Examples | GPTBot, CCBot | OAI-SearchBot, PerplexityBot, ClaudeBot |
| Block? | Your call — protects against training use | Blocking = invisible in AI answers |
A site owner sees “GPTBot” in their logs, adds Allow: / for GPTBot, and assumes they’ve fixed ChatGPT visibility. They haven’t. OAI-SearchBot is the retrieval crawler that drives ChatGPT Search citations. The two operate independently, and thousands of crypto sites have allowed one while blocking the other.
Finding 1: 68% of Crypto Sites Block AI Crawlers Without Knowing It
68% of sites in our dataset block at least one AI retrieval crawler. The blocking typically isn’t a deliberate choice.
One caveat before the mechanics: robots.txt is a voluntary compliance protocol, not a technical barrier. Major AI providers commit to respecting it, and the compliant bots do. Perplexity was documented in August 2025 running undeclared crawlers that rotate user-agents to bypass directives. Real-time user-directed fetches often bypass robots.txt entirely. For the mainstream bots driving most citations, though, robots.txt still matters — the problem is that most crypto sites are blocked even for bots that play by the rules.
Three blocking mechanisms dominate the data:
robots.txt wildcard disallow — Disallow: / under User-agent: * blocks every crawler. Some CMS security plugins generate this as a default, leaving site owners to add explicit exceptions they never add.
Cloudflare WAF rules — WAF evaluates requests before robots.txt is processed. A blanket “block all bots” rule silences AI crawlers completely even with clean robots.txt directives — those directives are simply never reached. This is the mechanism behind 61% of the blocked sites: not policy, but a WAF configured during setup and never revisited. It’s the most important distinction in the dataset: these sites didn’t choose to block AI crawlers. They just never explicitly chose to allow them, and the default made the choice for them.
Security plugin defaults — WordPress-based crypto blogs frequently run security plugins that add aggressive bot-blocking rules to protect against credential stuffing and spam, with no distinction between malicious scrapers and AI retrieval crawlers.
The GPTBot/OAI-SearchBot trap is the most common partial-fix error: allowing GPTBot (training) without adding OAI-SearchBot (ChatGPT Search retrieval). A site can have Allow: / for GPTBot and still be invisible in ChatGPT Search because OAI-SearchBot hits a WAF block first.
Quick manual check: open yoursite.com/robots.txt and look for User-agent: * with Disallow: / and no subsequent Allow directives for retrieval bots. That covers robots.txt — it won’t surface WAF-level blocking. The AI Visibility Checker audits both layers in 30 seconds.
One important limit: open crawler access is a prerequisite for citations, not a guarantee of them. It means AI bots can reach your pages — not that they’ll cite them. Content authority and entity clarity still determine what gets cited once access is open. Fixing access first is the only order that produces results; building authority while blocked produces none.
What it means for your business: Being in the 68% isn’t just missing citations — it’s missing the consideration stage entirely. AI didn’t evaluate your project and rank it low. It never saw it. For early-stage projects, that means launch content and protocol docs enter no AI knowledge layer regardless of quality. For established ones, a competitor with weaker fundamentals but open crawlers is getting cited in your place. Fixing WAF and robots.txt has zero content dependency and a measurable outcome within 48–72 hours.
Finding 2: Schema Markup Doesn’t Drive AI Citations
54% of sites in the bottom half of our dataset have zero JSON-LD schema. That gap in technical maturity is real. Its significance for AI citation rate, however, is not what most guides claim.
In May 2026, Ahrefs tracked 1,885 pages that added JSON-LD schema between August 2025 and March 2026, matching each against control pages with similar citation history. The result: no meaningful uplift on Google AI Mode (+2.4%), ChatGPT (+2.2%), or AI Overviews (−4.6%). The first two are statistically indistinguishable from noise.
A controlled experiment explains the mechanism. Researcher Mark Williams-Cook embedded an address exclusively inside invalid, made-up JSON-LD — nothing in visible page content. ChatGPT and Perplexity both extracted and returned it correctly. LLMs tokenize the <script> block as raw text. They read the words inside, but the schema vocabulary and structure are invisible to them. On top of this, Google deprecated FAQ rich results on May 7, 2026, removing the primary mechanism by which FAQPage schema was supposed to feed AI answer extraction.
Schema still has value — for classical Google and Bing ranking. Microsoft confirmed in March 2025 that schema helps Copilot understand content. ChatGPT Search draws largely from Bing’s index, so better classical ranking does translate upstream into AI citation, just indirectly. Organization schema on the homepage is worth keeping for entity clarity. Article schema across content pages is reasonable SEO hygiene. FAQPage is no longer worth prioritising.
| Schema Type | Actual Value | Where |
|---|---|---|
| Organization | Entity clarity → indirect AI benefit via classical SEO | Homepage |
| Article/BlogPosting | Classical SEO freshness signals | Content pages |
| FAQPage | Deprecated May 2026; no AI citation value | — |
What it means for your business: Stop treating schema as an AI visibility fix. If you’ve been deferring WAF and robots.txt work because “we’re still finishing the schema layer,” reverse that priority. A Cloudflare rule change that takes 20 minutes will produce more AI citation lift than a schema overhaul quoted at 20–40 hours. Do Organization schema because it’s fast and supports classical SEO — then move to the fixes that actually move the AI needle.
Finding 3: Domain Rating and AI Visibility Are Not Correlated
The dominant narrative in crypto SEO is straightforward: build links, grow authority, get cited by AI. The data doesn’t support this.
In our 500+ site dataset, Domain Rating above 40 showed no consistent improvement in AI Visibility Score once crawler access gaps were accounted for.
High-DR sites underperform for predictable reasons. Cloudflare WAF blocking applied during a past security incident and never revisited. React or Next.js SPAs without server-side rendering — AI crawlers don’t execute JavaScript, so they see a blank page regardless of how many backlinks point at the domain. A DR 60 exchange built on a React SPA with no SSR has zero AI-readable content on its product pages. The backlinks don’t help. Crawlers land, see nothing, leave.
Low-DR sites that outperform their peer group share a consistent profile: open crawlers, server-rendered HTML, a clean entity page. Technical hygiene compensates for authority deficit at the niche query level. For category-level queries like “best crypto exchange,” authority still dominates. For specific niche queries — “non-KYC Solana DEX with low fees” — a technically clean small site can appear in AI answers above an established competitor that blocks PerplexityBot. The window where technical hygiene creates this advantage exists now and is probably finite as the space matures — established players will eventually fix their configurations.
What it means for your business: For established projects, the finding is a warning — your DR isn’t protecting your AI visibility. If your WAF blocks retrieval crawlers, a newer competitor with clean configuration is getting cited for queries where you should be the obvious answer. For early-stage projects, it’s an opening. Fix the four technical categories and you’re competing on equal footing with established players for niche queries today, not after 18 months of link building.
What the Top 10% Have in Common
Five patterns appear across every high-scoring site in the dataset:
1. Named robots.txt directives for every major retrieval bot — not Allow: * but explicit directives for OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Claude-SearchBot, and Google-Extended.
2. Baseline Organization schema on the homepage — the one schema type that meaningfully helps entity resolution in search engines, which indirectly feeds AI citation. Given the Ahrefs data, this pattern in top sites likely reflects general technical maturity rather than schema being the causal factor.
3. An llms.txt file — under 4% of the general dataset has one; every top-10% site does. Current evidence puts its citation impact as negligible for ChatGPT and Perplexity search; its real value is in agentic contexts (IDE agents, MCP integrations). Deploy it anyway — it costs nothing and positions you if major AI search engines adopt it.
4. Server-side rendered HTML on all public pages — for React/Next.js SPA sites without SSR, this is the highest-priority fix. AI crawlers don’t execute JavaScript, so they see a blank page regardless of everything else. A DR 60 DeFi protocol built on a client-side React app is, from an AI crawler’s perspective, essentially an empty domain.
5. A clean entity definition page — who the organisation is, what it does, what chain or protocol it runs on, when it was founded. Without it, AI systems infer identity from fragmented signals across multiple pages, which produces inconsistent citations or none. A clear, crawlable About page structured to answer “what is X” directly is enough.
What these sites don’t have: exceptional Domain Ratings. The correlation is configuration, not authority.
Fix Your Score in One Day
Run the AI Visibility Checker before starting. It shows which of these steps your site actually needs and weights them by impact — no point spending four hours on schema when your WAF is silently blocking every AI crawler that tries to reach you.
Step 1 — Update robots.txt (30 min)
Add explicit Allow directives for each of the following user-agents, placed above any Disallow rules already in the file: OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Claude-SearchBot, and Google-Extended. Each bot needs its own User-agent block with Allow: / — a single wildcard isn’t reliable enough here. GPTBot can stay blocked if you don’t want OpenAI training on your content; blocking it has no effect on ChatGPT Search.
Step 2 — Check Cloudflare WAF (20 min)
Security → WAF → Custom Rules. Look for any rule blocking bot traffic by bot score or user-agent. Step 1 changes nothing if a WAF rule fires first — directives in robots.txt are never reached if the request is dropped at the WAF layer.
Step 3 — Create llms.txt (20 min)
Create a plain text file at yoursite.com/llms.txt. The minimum viable version: one line identifying the project, two to three sentences describing what the site contains and who it’s for, then a short list of key page URLs with a one-line description of each. The goal is to give AI agents a direct, unambiguous description of your entity — homepage, docs, about page at minimum.
Step 4 — Add baseline schema (1–2 hours)
Organization schema on your homepage, Article/BlogPosting on content pages. Skip FAQPage — deprecated by Google in May 2026 and ignored structurally by AI systems anyway. Schema matters for classical SEO, not for direct AI citations.
Step 5 — Verify HTML renders without JavaScript
Open your key pages with JavaScript disabled in your browser’s dev tools, or fetch them directly via curl and inspect the response. Blank or near-empty HTML means SSR is the blocker — fix that before anything else on those pages.
What 500+ Audits Tell Us
Crypto’s AI visibility problem is a configuration problem. Crawler access and server-side rendering are the two variables that separate cited sites from invisible ones. Schema and llms.txt matter at the margins, and the industry has overclaimed their impact — the May 2026 Ahrefs data settled the schema debate.
The gap between the top 10% and the rest is closing for projects that fix the configuration layer now. For those waiting on a schema audit to move the needle, technically clean competitors are already getting cited on the back of a WAF rule change and 30 minutes of work.
Frequently Asked Questions (FAQ)
GPTBot feeds OpenAI training data. OAI-SearchBot powers ChatGPT Search citations in real time. They’re separate crawlers with separate user-agent strings that need separate robots.txt directives. You can block GPTBot while allowing OAI-SearchBot, and vice versa — the two operations have no effect on each other.
No. Blocking GPTBot removes your content from OpenAI’s training pipeline. OAI-SearchBot is what drives ChatGPT Search citations, and it operates independently. Many sites in our dataset block GPTBot deliberately — to protect content from training use — and still achieve high AI Visibility Scores.
No. DR above 40 shows no consistent correlation with AI Visibility Score in our dataset once crawler access is controlled for. A DR 70 React SPA with no SSR scores zero — same as a DR 10 site with the same configuration.
Yes — the AI Visibility Checker audits all four categories in 30 seconds, no account needed.
Almost certainly a WAF rule firing before robots.txt, or JavaScript-rendered pages returning blank HTML to crawlers. Schema is unlikely to be the cause — the May 2026 Ahrefs study found no meaningful citation uplift from adding JSON-LD.
Share with
Rate the article
