CiteFuelBot

CiteFuelBot is the crawler CiteFuel uses to run audits that site owners request. If you saw it in your logs, someone submitted your domain for an AI visibility audit, or your domain was part of our published research sampling.

Identification

User-Agent: CiteFuelBot/1.0 (+https://citefuel.com/bot)

What it fetches

• /robots.txt, /llms.txt, your sitemap, and your homepage — the publicly accessible configuration surface.
• For paid multi-page audits: up to 25 pages the requester specifies, all public URLs.
• During an audit we also send a small number of requests with documented AI-crawler user-agent strings (GPTBot, ClaudeBot, etc.) to test whether your WAF blocks them — ≤6 requests, homepage only.

Limits & compliance

• Rate limit: maximum 1 request/second per host; typical audits make 10-40 requests total.
• No spidering: we do not recursively crawl beyond the pages listed above.
• No login areas, no form submissions, no personal data collection.
• Research crawls fetch only root-path configuration files, never page content at scale.

How to opt out

Add this to your robots.txt and CiteFuelBot will not audit your site:

User-agent: CiteFuelBot
Disallow: /

Or email hello@citefuel.com and we'll add your domain to the permanent exclusion list within 2 business days.