Robots.txt for AI Crawlers & Product Tokens: 2026 Policy Guide
Robots.txt publishes access and data-use policy for multiple token classes: training crawlers, retrieval crawlers, user-triggered fetchers, and non-requesting product-control tokens. Most robots.txt files were written before any of these existed, which means most sites are running a policy they never chose. Here's the full token list, what each one actually controls, and three reviewable policy examples for different publisher intents.
The 14 tokens, and what each one feeds
| Token | Operator | What it feeds |
|---|---|---|
GPTBot | OpenAI | ChatGPT training index |
OAI-SearchBot | OpenAI | ChatGPT search retrieval |
ChatGPT-User | OpenAI | Live browsing when a user asks for your page |
ClaudeBot | Anthropic | Claude training index |
Claude-SearchBot | Anthropic | Claude search retrieval |
anthropic-ai | Anthropic | Legacy token — keep rules consistent with ClaudeBot |
Google-Extended | Non-requesting Gemini Apps / Vertex AI product-control token; no Search ranking effect | |
PerplexityBot | Perplexity | Perplexity search index |
Perplexity-User | Perplexity | Live page visits on user request |
Meta-ExternalAgent | Meta | Meta AI training and retrieval |
Applebot-Extended | Apple | Non-requesting Apple Intelligence data-use control; Applebot performs requests |
CCBot | Common Crawl | Open corpus used by many AI labs |
Bytespider | ByteDance | ByteDance AI crawler |
YouBot | You.com | You.com AI search |
Two structural details hide in that table. First, OpenAI and Anthropic split training and
retrieval into separate tokens — you can be absent from a model's baked-in knowledge yet still
be cited in live answers, or vice versa. Second, Google-Extended is a control token, not a
separate crawler: ordinary Googlebot does the fetching, and Google-Extended only governs whether that
content can support eligible Gemini product uses. Applebot-Extended is likewise a data-use control read
by Applebot, not a separate crawler. Google-Extended does not affect Search inclusion or ranking,
including AI Overviews and AI Mode.
The decision: what does blocking each class cost?
- Disallow training crawlers (for example GPTBot, ClaudeBot, and CCBot) to state that those agents should not fetch affected paths for their documented data uses. Downstream model behavior is not measured by the rule itself.
- Disallow retrieval crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and the named crawler should not fetch affected paths directly. The rule does not prove absence from every answer, because engines can use other indexes and sources.
- Disallow user-triggered fetchers (ChatGPT-User, Perplexity-User) to decline that documented on-demand access path. This is a publisher policy choice, not a ranking lever.
- Set product-control tokens (Google-Extended, Applebot-Extended) according to data-use intent; neither token makes its own HTTP requests.
Three reviewable policy examples
1. Explicitly allow selected requesting crawlers
Use only after confirming that retrieval, training, and user-request access match your policy. Allow rules do not guarantee visibility:
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /2. "Cite me, don't train on me"
State separate retrieval and training intent. This permits documented retrieval access but does not guarantee live citation:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /Honest caveat: the training/retrieval boundary is operator policy, not physics. It holds today for the operators that document it; it is not a cryptographic guarantee.
3. Block everything
For a broad restrictive policy, name the relevant tokens and verify effective rules. This does not prove removal from every AI surface:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
# …repeat for each of the 14 tokens aboveThe three configurations that block silently
- Wildcard fallback. A
User-agent: */Disallow: /group written years ago applies to every AI crawler that lacks its own group — robots.txt semantics fall back to the wildcard. Your file blocks GPTBot without containing the string "GPTBot". - Allowlist files. A robots.txt that names Googlebot and Bingbot and disallows the rest was a reasonable 2019 decision and a 2026 AI blackout.
- The WAF layer. Bot managers (Cloudflare Bot Fight Mode is the classic) challenge AI user-agents before robots.txt is ever read. Your robots.txt says "allowed," the firewall serves a 403. This is the most common failure we find on otherwise well-configured sites, and no robots.txt inspection can see it — compare responses using documented user-agent labels.
The free AI crawler checker parses your live robots.txt with effective-rule semantics (wildcard fallback included) against all 14 tokens. It generates an Allow suggestion only after you select an explicit policy intent. The full audit also requests your homepage using documented user-agent labels to catch response differences. These are labeled test requests, not traffic from verified crawler networks.
Robots.txt is policy, not enforcement
A robots.txt line is a request that well-run crawlers honor — the major labs document compliance and observably follow it — but it is not a wall. If you must hard-block, enforce at the WAF. And the complement holds too: robots.txt grants access but explains nothing. That's the job of optional llms.txt proposal — a service may choose whether to consume it, and Google Search ignores it. Access first, useful visible content second; the generator builds the second part from your sitemap in about forty seconds.