CiteFuelBot

CiteFuelBot is the crawler CiteFuel uses for on-demand audits. If you saw it in your logs, a public URL on your domain was submitted to CiteFuel for an audit. Submission does not prove that the submitter owns or represents the domain.

Identification

User-Agent: CiteFuelBot/1.0 (+https://citefuel.com/bot)

What it fetches

  • /robots.txt, /llms.txt, a discovered sitemap, the submitted page, and other public URLs required by the selected audit checks.
  • For paid multi-page audits: up to 25 pages the requester specifies, all public URLs.
  • In addition to the normal homepage fetch, the access/WAF heuristic sends seven visibility-relevant homepage probes and may send up to nine deep-path probes. Training and product-control tokens are reported as policy choices but do not lower Readiness solely because they are blocked. These are CiteFuel configuration probes, not traffic from the named providers or their verified crawler networks.

Limits & compliance

  • Concurrency: no more than 20 in-flight requests across an audit and no more than 4 in-flight requests to one host, with bounded request and page budgets.
  • No open-ended spidering: requests stay within the submitted audit scope, sitemap/page limits, and configured budgets.
  • No authentication or form submission. The audit requests public URLs and processes only the response content needed for the published checks.
  • Research crawls fetch only root-path configuration files, never page content at scale.

Crawler concerns and exclusion requests

The audit reads and reports robots.txt directives, but the current runner does not create a permanent domain exclusion from User-agent: CiteFuelBot and Disallow: /. We do not represent that directive as an automatic or instant opt-out. To report unexpected traffic or request exclusion from future on-demand audits, use the contact form and select Crawler / CiteFuelBot, or email support@citefuel.com. Include the domain, affected URL paths, timestamps, relevant log lines, and the action requested. A human reviews the request and will confirm what can be applied; submission alone is not confirmation that an exclusion is active.