What is ClaudeBot?
ClaudeBot is Anthropic's web crawler, identified by the user-agent string ClaudeBot, used to collect publicly available content for training and updating Claude AI models. A separate user-agent, Claude-SearchBot, handles live retrieval for Claude's web-connected responses.
Like GPTBot, ClaudeBot's access is governed by standard robots.txt Disallow directives. Sites blocking ClaudeBot prevent Anthropic from using their content in training data, but should separately configure Claude-SearchBot if they want to appear in Claude's cited sources.
As of 2026, Anthropic documents both user-agents and respects robots.txt exclusions. CiteFuel tests both ClaudeBot and Claude-SearchBot separately, since many sites accidentally block retrieval access while intending to block only training access.