What is AI Crawlability?
AI crawlability is the degree to which AI systems — ChatGPT, Claude, Perplexity, Google Gemini — can technically access, fetch, and parse a website's content. It is the foundation layer of AI search visibility: a site with perfect content and zero AI crawlability is invisible to AI answers.
AI crawlability is controlled at four layers: (1) robots.txt directives for AI user-agents like GPTBot and ClaudeBot, (2) WAF and bot-management rules that may challenge AI crawlers before robots.txt is even read (Cloudflare Bot Fight Mode is a common accidental blocker), (3) server-level blocks by user-agent or ASN, and (4) rendering — content that only exists after heavy client-side JavaScript may be partially invisible to crawlers that don't execute scripts.
The most common AI crawlability failure is a legacy robots.txt written before AI crawlers existed: a wildcard Disallow or an allowlist that only names Googlebot silently blocks every AI agent. CiteFuel's audit probes your site with real AI crawler user-agent strings to detect both robots.txt and WAF-layer blocks.