How to Manage AI Search and Retrieval Crawlers in Robots.txt

Why It Matters

AI search engines rely on specialized retrieval bots to ground generated answers in real-time web facts. Blocking these crawlers prevents your content from appearing as cited primary sources in conversational AI search results.

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

Strategic SEO & GEO Impact

Allowing AI retrieval crawlers ensures your published research, product pages, and brand facts can be fetched live to generate grounded user responses with high-value organic referral links.

Which Bots Power Which Answers

OAI-SearchBot gathers pages for ChatGPT search, Claude-SearchBot for Claude, and PerplexityBot for Perplexity. Google’s AI Overviews and AI Mode use the regular Googlebot, and Microsoft Copilot uses Bingbot, so being open to normal search crawlers matters for AI answers too.