Docs & playbooks
Robots.txt for AI crawlers: Allow major bots with confidence
Documented directives for GPTBot, PerplexityBot, CCBot, Google-Extended, and Anthropic bots—paired with privacy tips.
Quick copy
Allow major AI crawlers (example)
Verify policies with your legal and security teams before publishing.
- User-agent: GPTBot — Allow: /
- User-agent: PerplexityBot — Allow: /
- User-agent: CCBot — Allow: /
- User-agent: Google-Extended — Allow: /
- User-agent: ClaudeBot — Allow: /
Notes
Best practices
Balance access with governance and privacy needs.
- Publish a clear privacy policy and data retention statement.
- Use allowlists for public content; restrict PII and private endpoints.
- Monitor crawler activity and error rates after changes.
Related resources
Pair robots with structure
Let the right bots in and give them clean data.