Reference · Updated June 2026
AI Crawler & Product-Control Tokens: 2026 robots.txt Reference
This reference covers requesting AI crawler user agents and non-requesting product-control tokens. Use these strings when writing
User-agent: blocks in robots.txt. Every entry is verified against the
official bot documentation of the respective company, where one is published.
This page tracks 14 distinct AI-system robots tokens. They include training crawlers, retrieval crawlers, user-triggered fetchers, and non-requesting product-control tokens such as Google-Extended and Applebot-Extended. Your robots.txt strategy should treat these differently — you may want to allow retrieval while blocking training.
Our June 2026 crawl of 10,000 top sites found that 12.5% published a robots.txt disallow rule affecting at least one tracked crawler or product-control token, and 10.5% named at least one tracked token explicitly. GPTBot had the highest disallow rate (11.3%), followed by ClaudeBot (10.3%) and Google-Extended (9.5%); the last is policy configuration, not crawler blocking.
Complete User Agent Reference
| User-agent token | Company | Type | Documentation |
|---|---|---|---|
GPTBot | OpenAI | Training | platform.openai.com/docs/bots |
ChatGPT-User | OpenAI | Retrieval (browsing) | platform.openai.com/docs/bots |
OAI-SearchBot | OpenAI | Search index | platform.openai.com/docs/bots |
ClaudeBot | Anthropic | Training + retrieval | support.anthropic.com |
anthropic-ai | Anthropic | Legacy token | support.anthropic.com |
Claude-SearchBot | Anthropic | Retrieval | support.anthropic.com |
Google-Extended | Non-requesting Gemini Apps / Vertex AI product-control token; no Search ranking effect | developers.google.com | |
Googlebot | Search + AI Overviews | developers.google.com | |
PerplexityBot | Perplexity | Index + retrieval | docs.perplexity.ai/guides/bots |
Applebot-Extended | Apple | Non-requesting Apple Intelligence data-use control; Applebot performs requests | support.apple.com |
Applebot | Apple | Spotlight / Siri | support.apple.com |
YouBot | You.com | Index + retrieval | about.you.com |
meta-externalagent | Meta | Training (Meta AI) | developers.facebook.com |
Bytespider | ByteDance | Training (Doubao / TikTok AI) | No public documentation |
Reviewable robots.txt policy examples
Allow all tracked requesting crawlers and product tokens (default — no entry needed)
If you want all AI crawlers to have full access, you don't need to add anything to robots.txt. The default is allow. Only add rules if you want to restrict.
Block all AI training crawlers
User-agent: GPTBot
User-agent: Google-Extended
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Bytespider
Disallow: /Allow retrieval, block training
# Block training crawlers
User-agent: GPTBot
User-agent: Google-Extended
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Bytespider
Disallow: /
# Allow retrieval crawlers (AI answers, Perplexity, etc.)
User-agent: ChatGPT-User
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Claude-SearchBot
Allow: /For a complete worked example with explanations, see the robots.txt for AI crawlers guide. To inspect how your current robots.txt treats these tokens, run the free policy checker — it reports all 14 and only generates Allow suggestions after you choose an explicit policy intent.
How to verify your robots.txt is working
After editing your robots.txt, run a CiteFuel audit. The check panel shows
the effective rule result for each crawler — whether it's allowed, disallowed, or missing
from your file. The audit also flags wildcard catches (User-agent: * with
Disallow: /) that disallow crawler tokens even when the file does not name them explicitly.
You can also generate an optional llms.txt proposal-format draft for services that choose to consume it. It does not override crawler access rules or guarantee prioritization.
Frequently asked questions
Why are there two user agents for OpenAI (GPTBot and ChatGPT-User)?
GPTBot is the training crawler — it fetches content to train ChatGPT models. ChatGPT-User is the retrieval crawler — it fetches live content when a user asks ChatGPT a question with browsing enabled. You can allow one and block the other. Most sites that care about AI citations allow ChatGPT-User and block GPTBot (no training benefit to them).
Does blocking GPTBot prevent my site from appearing in ChatGPT answers?
If ChatGPT-User is allowed (or not listed at all in your robots.txt), your site can still appear in ChatGPT browsing answers. GPTBot is the training crawler — blocking it opts you out of training data, not live answers. Always be explicit: list ChatGPT-User as an allow group separately from GPTBot.
What is anthropic-ai vs ClaudeBot?
Both belong to Anthropic. ClaudeBot is the primary Anthropic crawler. anthropic-ai is an older token — some Claude systems use it. For full coverage, block (or allow) both. The robots.txt guide shows both in the same User-agent group.
Should I use Disallow: / or specific path blocks?
Disallow: / blocks the entire site. If you only want to block specific paths from AI crawlers (say, member-only content), use path-specific rules. If you want to opt out of AI training entirely, Disallow: / under the training bot groups is the correct approach.