AI Crawler

AI crawlers split into two purposes: training crawlers (build the corpus used to train the next model version) and retrieval crawlers (fetch fresh content at query time for RAG-style responses). Perplexity, ChatGPT browsing, and Gemini all use retrieval crawlers; OpenAI and Anthropic also operate training crawlers.

Identify them by user agent and respect their rate limits. Block selectively if needed, but understand the trade-off: blocking GPTBot means future ChatGPT models will not have your content.

Related terms

• robots.txt — A plain-text file at /robots.txt that tells web crawlers (search engines and AI bots) which pages they may access. Mis-configuring it is the #1 way brands accidentally block themselves from AI engines.
• llms.txt — A plain-text file at the root of a domain (like /robots.txt) that gives AI systems a curated, machine-readable summary of your brand, products, and key documentation links.
• Answer Engine — An AI system that responds to natural-language queries with a synthesized answer rather than a list of links — examples include ChatGPT, Perplexity, Google Gemini, Microsoft Copilot, and Anthropic Claude.