Anthropic Splits Claude Crawlers Into Three Bots – What Publishers Need to Know

Anthropic has updated its crawler documentation, separating its bots into three distinct categories:

• ClaudeBot (training),
• Claude-SearchBot (search indexing), and
• Claude-User (user-initiated retrieval).

The key takeaway for publishers: blocking ClaudeBot no longer blocks the others.

Anthropic warns that restricting Claude-SearchBot may reduce your site’s visibility in AI-powered search results.

This mirrors OpenAI’s structure and signals a broader industry shift.

With 71% of top news sites already blocking at least one retrieval bot, many are unknowingly sacrificing AI search visibility.

It’s time to invest a few minutes in checking and improving your robots.txt file.