How to Control AI Model Training Bots in Robots.txt

Why It Matters

Controlling AI training scrapers allows organizations to protect proprietary content and intellectual property. Disallowing training scrapers provides a legitimate mechanism to manage data licensing rights.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

Important Crawler Distinction

Blocking training crawlers (e.g., GPTBot) does not block retrieval search bots (e.g., OAI-SearchBot) from featuring your site in live conversational AI search answers.

What Google-Extended and Applebot-Extended Do

These aren’t separate crawlers but tokens that tell Google and Apple not to use your content for training their AI models (Gemini and Apple Intelligence). Blocking them doesn’t affect your rankings in Google Search or your appearance in Siri and Spotlight.

A Request, Not a Lock

robots.txt is followed voluntarily. Reputable AI companies honor it, but it can’t stop scrapers that ignore it; for that you need your CDN or firewall’s bot controls.