How to Configure WAF and CDN Firewalls to Allow Legitimate AI Crawlers

Why It Matters

If your WAF presents JavaScript challenges or 403 errors to automated bots, AI search crawlers cannot index your content, eliminating your site from AI search citations.

• Enable CDN AI Bot Directory: Enable built-in verified "AI Bots" rules in modern CDNs (such as Cloudflare's verified bot directory).

• Bypass Security Challenges for Verified Bots: Ensure security rules do not present CAPTCHAs to requests from verified search engine IP ranges.

• Monitor Bot Block Logs: Review firewall block logs weekly to verify legitimate AI retrieval crawlers are not encountering HTTP 403 or 503 error codes.

Verify, Don’t Trust the Name

Anyone can send a request claiming to be GPTBot or ClaudeBot. OpenAI, Anthropic, Perplexity, Google and Microsoft publish their crawler IP ranges, and CDN “verified bot” features check them for you. Allowing a bot by user-agent alone opens a gap scrapers can use.