The State of AI Crawlability 2026:
We Crawled 10,000 Popularity-Ranked Domains

The five-signal configuration distribution is wide

In June 2026, CiteFuel crawled 10,000 domains from Tranco ranks 1,001–11,000 — a large popularity-ranked cohort, not a statistically representative sample of the open web — to inventory five crawl and page-configuration signals. Of the 10,000 domains queried, 7,543 returned parseable homepages. The rest timed out, returned non-HTML roots, or disallowed our crawler (217 domains honored; see Methodology).

The measured conclusion is narrower: most sites did not publish every item in CiteFuel's five-signal configuration checklist. This crawl did not measure rankings, recommendations, citations, or traffic, so a low checklist score is not proof that a site is invisible to AI.

Chart 1: Five-signal configuration index (0–5)

Each reachable site was scored on five binary signals: llms.txt present, no disallow rule for the tracked crawler/product tokens, Organization schema present, canonical tag present, and HTTPS redirect active. Scores range from 0 (none) to 5 (all five).

Chart 2: Per-token robots.txt disallow rates

We checked every domain's robots.txt against five crawler or product-control tokens using effective-rule semantics — including wildcard fallback. A token is counted as disallowed when the effective rule is Disallow: / for that user-agent, whether the token is named explicitly or inherited from a wildcard group. Google-Extended is different from the request-capable crawlers in this chart: Google documents it as a standalone product token that does not make HTTP requests. Its setting does not affect inclusion or ranking in Google Search, including AI Overviews and AI Mode.

The silent blocking problem

Across the sample, 1.9 percentage points had a wildcard rule that disallowed tracked tokens without naming those tokens explicitly. For example, GPTBot can inherit User-agent: * and Disallow: / even when the string "GPTBot" does not appear in the file. This is an effective-rule observation, not evidence about the administrator's intent.

Chart 3: Adoption by rank decile — the counterintuitive curve

The conventional assumption is that the most popular sites are the most technically sophisticated, and therefore the first to adopt emerging standards. For llms.txt, this is wrong.

The practical implication: llms.txt is currently a deliberate choice, not a default. Mid-tier sites with engaged engineering teams are adopting it faster than the media conglomerates and legacy institutions that dominate the top 1,000. The crawl did not test whether any provider consumes or prefers llms.txt; Google Search explicitly ignores it.

Chart 4: Signal adoption rates across the web

AI retrieval systems layer multiple signals. We measured five: HTTPS redirect (the baseline), canonical tag, sitemap declaration in robots.txt, Organization schema, and llms.txt. Each sits at a different maturity point on the adoption curve.

Chart 5: llms.txt presence was associated with a 20% lower median TTFB

Across all 7,543 reachable sites, we recorded the Time to First Byte (TTFB) of the homepage response. The distribution splits sharply by llms.txt status.

Configuration index by industry vertical

Industry classification uses TLD heuristics applied at crawl time (same as crawl-study/analysis.py). Only verticals with n ≥ 200 domains are published to ensure statistical credibility. Two specific verticals clear the bar; the general-web bucket (n = 8,837) serves as the baseline for all-industry comparisons.

Verticals with n < 200 (education n=164, government n=172, media n=93, e-commerce n=5) are omitted — insufficient sample for reliable inference. Classification: TLD heuristics (gov/mil → government; edu/ac → education; org/ngo → nonprofit; io/dev/app/ai/tech → tech; news/media/tv → media; shop/store → e-commerce; all others → general). Reproducible: python3 crawl-study/benchmark_verticals.py.

The downloadable study data lets researchers reproduce the five-signal comparisons shown here. The current CiteFuel checker uses a broader methodology and reports its own evidence coverage; its headline is not the same metric as this historical configuration index.

Methodology

Full transparency on what we measured, how we measured it, and where the data falls short — because credibility depends on it.

Sample frame

Tranco ranks 1,001–11,000 (10,000 domains), June 2026 list. Ranks 1–1,000 excluded to avoid sampling bias from mega-platforms with outlier infrastructure.

Crawl date

June 12, 2026. Point-in-time snapshot. All rates reflect conditions on that date. Re-runs are planned quarterly.

Crawler

CiteFuelBot/1.0. Full robots.txt compliance. Fetched homepage only (root path). Not a training crawler.

Reachability

7,543 / 10,000 homepages returned parseable HTML. Remainder: timeouts, connection errors, non-HTML roots. All 10,000 are in the denominator for blocking-rate calculations.

Robots.txt opt-outs

217 domains disallowed CiteFuelBot. We honored the opt-out and did not fetch their homepage. These are recorded as robots_excluded in the raw data.

Configuration index

5 binary signals: (1) llms.txt present, (2) no disallow rule for the five tracked crawler/product tokens, (3) Organization schema present, (4) canonical tag present, (5) HTTPS redirect active. Each is 1 point. This is not a ranking or visibility score.

Blocking detection

Full effective-rule semantics including wildcard fallback (User-agent: *). Rates computed over all 10,000 domains, not just reachable ones.

TTFB

Median homepage response time recorded during the same crawl. Computed over reachable sites only. Network conditions affect absolute values; the relative gap is the signal.

Limitations

• Homepage-only. We crawled one URL per domain. Inner pages and subdomains are not captured, so the configuration index cannot characterize the whole site.
• Point-in-time. All rates are a single-date snapshot. The web changes daily. We publish the crawl date prominently and plan quarterly re-runs.
• No WAF probing. We measured effective robots.txt disallow rules, not whether requests from verified crawler networks would be accepted at the CDN/WAF layer. WAF behavior is outside this dataset and no total enforcement rate can be inferred from it.
• Industry breakdown is directional. Industry classification uses TLD heuristics (Tranco carries no category data). Sub-bucket rates (tech, edu, gov) are directional, not authoritative.
• Tranco list dynamics. Tranco is a popularity aggregate, not a curated quality list. Very new domains and CDN edge nodes may appear at these ranks.

Data and reproducibility

The aggregate JSON (stats-2026-06-12.json) is published alongside this report. The full per-domain CSV (10,000 rows) is available on request at research@citefuel.com — we share it freely with journalists, researchers, and developers. The crawl harness code is likewise available on request at the same address.

Publication standards, authorship and review responsibilities, AI-assistance disclosure, funding controls, and conflict handling are documented in CiteFuel's editorial policy. Questions about a figure or interpretation can be submitted through the corrections process.

What this means for site owners

This study is a configuration inventory, not a ranking experiment. Site owners can use it to review intentional robots policies, canonical and HTTPS basics, and whether structured data truthfully matches visible content. Those checks can remove technical ambiguity, but the study provides no evidence that adopting llms.txt or schema will cause an answer engine to mention or cite a site.

For Google specifically, normal indexing and snippet eligibility remain the foundation for AI features; no special AI schema is required, and Google says it ignores llms.txt. Any robots.txt or structured-data change should reflect the site's actual policy and visible content, then be tested after deployment.

How to cite this study

CiteFuel. "The State of AI Crawlability 2026: We Crawled 10,000 Popularity-Ranked Domains."
CiteFuel Research, June 12, 2026. https://citefuel.com/research/state-of-ai-crawlability-2026

@techreport{citefuel2026crawlability,
  author       = {CiteFuel},
title        = {The State of AI Crawlability 2026: We Crawled 10,000 Popularity-Ranked Domains},
  institution  = {CiteFuel},
  year         = {2026},
  month        = {jun},
  type         = {Research Report},
  url          = {https://citefuel.com/research/state-of-ai-crawlability-2026},
  note         = {Data: https://citefuel.com/research/data/stats-2026-06-12.json}
}