List of Crawlers and structure of robots.txt for better AI-Crawlers readability
How should your robots.txt should be structured:
Platform - Crawlers / Agents
1. ChatGPT - GPTBot, OAI-SearchBot, ChatGPT-User
2. Google AI - Googlebot (various), Google-Extended, Gemini-Deep-Research
3. Gemini (Same as Google AI) - (Googlebot, Google-Extended, Gemini-Deep-Research)
4. Claude - ClaudeBot, Claude-User, Claude-SearchBot, Claude-Web
5. Grok - GrokBot, xAI-Grok, Grok-DeepSearch
6. Perplexity - PerplexityBot, Perplexity-User Layout of Robots.txt
Robots.txt
# ============================
# OpenAI / ChatGPT Crawlers
# ============================
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
# ============================
# Google / Gemini Crawlers
# ============================
User-agent: Googlebot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: Gemini-Deep-Research
Allow: /
# ============================
# Anthropic / Claude Crawlers
# ============================
User-agent: ClaudeBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-Web
Allow: /
# ============================
# xAI / Grok Crawlers
# ============================
User-agent: GrokBot
Allow: /
User-agent: xAI-Grok
Allow: /
User-agent: Grok-DeepSearch
Allow: /
# ============================
# Perplexity Crawlers
# ============================
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
# ============================
# Sitemap (keep at the end)
# ============================
Sitemap: https://www.example.com/sitemap.xml
By implementing these strategies, you can optimize your website for AI crawlers, enhancing the likelihood of your content being cited and gaining greater visibility through mentions.
Replies