What is AI Crawlability?
AI crawlability describes whether a particular crawler or retrieval service can directly fetch and parse a website's content. A block can prevent that direct request, but it does not prove that every product or model is unaware of the page through search indexes, partner data, or other sources.
AI crawlability is controlled at four layers: (1) robots.txt directives for AI user-agents like GPTBot and ClaudeBot, (2) WAF and bot-management rules that may challenge AI crawlers before robots.txt is even read (Cloudflare Bot Fight Mode is a common accidental blocker), (3) server-level blocks by user-agent or ASN, and (4) rendering — content that only exists after heavy client-side JavaScript may be partially invisible to crawlers that don't execute scripts.
A wildcard Disallow, explicit bot rule, or WAF challenge can block a documented crawler. CiteFuel sends requests labeled with documented crawler user-agent strings to compare responses; that is a configuration test, not proof that the request came from a verified crawler network.