AI Crawler Access Checker

This checker evaluates how your website's robots.txt treats documented crawler and product tokens. The list includes retrieval crawlers, training crawlers, user-triggered fetchers, and control tokens. They do not all perform the same job: AI operators distinguish training access (reading content to train models) and retrieval access (reading content to answer live queries). A blocked training crawler may still allow retrieval — or vice versa. Most robots.txt configurations were written before these tokens existed and may apply an unintended policy.

This tool fetches your live robots.txt, parses it against 14 tracked tokens, and returns a color-coded policy matrix: green (allowed by the file), red (disallowed by the file), yellow (no explicit named rule). A robots.txt rule states publisher policy; this check does not prove network-layer enforcement. The tool generates no Allow rules by default. If you explicitly request a suggestion, retrieval/user-request tokens remain separate from training and product-control tokens so a deliberate data-use policy is not silently reversed. Google-Extended and Applebot-Extended are non-requesting control tokens, not live crawlers.

No signup required. Paste your URL. Results in under 10 seconds.

How to use

1. Paste your site's root URL (e.g., https://yourdomain.com) into the field above.
2. Click "Check AI Crawler Access."
3. Review the policy matrix — a red row means the named crawler or token is disallowed by the published file, not that enforcement or every route to visibility was proven.
4. If you selected an output intent, review the generated suggestion against your licensing, privacy, training, and retrieval policy before changing anything.
5. Optionally: run a full CiteFuel audit to check llms.txt, schema, and passage citability alongside crawler access.

Related: llms.txt Validator · CiteFuel Readiness & Visibility Score · Methodology