Decide your policy for AI crawlers
Why it matters
This is a summary of which AI crawlers your robots.txt allows or blocks, not a fault. Whether to allow them is your choice, because asking training crawlers to stay away is a request that well-behaved ones honour, while blocking AI search crawlers can reduce how often you are cited in AI answers.
The fix
- Decide per purpose. Training crawlers (such as GPTBot, ClaudeBot, CCBot, and the Google-Extended and Applebot-Extended control tokens) collect content for models. Search crawlers (such as OAI-SearchBot and PerplexityBot) index pages so they can be cited in answers and can send visitors.
- User-triggered fetchers such as ChatGPT-User and Perplexity-User visit a page because a person asked about it. Their operators say robots.txt rules may not apply to them, so a Disallow for them is a request that may be ignored.
- To opt out, add a group with the crawler's product token and Disallow in robots.txt. Blocking Google-Extended does not change how Googlebot indexes you for Google Search.
- Remember that robots.txt is voluntary. Well-behaved crawlers follow it, others may not, so use access controls (a login, or bot rules at your CDN or firewall) if the content must be protected.
robots.txt
# Example opt-out from model training; keep or remove each group as you decide User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: / User-agent: Google-Extended Disallow: /
Related
- Reported by the AI crawler access rules check
Scan your domain to see whether this applies to you.