robots.txt

By 2025, the major AI crawlers each have their own user agent: GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot (Perplexity), Google-Extended (Gemini training), Bytespider (ByteDance/Doubao). Many sites block these by default after a 2024 wave of “protect our content” advice — which has the unintended effect of removing the brand from AI answers entirely.

For most B2B brands, you want these bots to crawl you. Add explicit User-agent: GPTBot / Allow: / rules to your robots.txt. VibecodeAEO audits this automatically.

Related terms

• llms.txt — A plain-text file at the root of a domain (like /robots.txt) that gives AI systems a curated, machine-readable summary of your brand, products, and key documentation links.
• AI Crawler — A web crawler operated by an AI company to gather content for training or for live answer-engine retrieval. Distinct from traditional search crawlers like Googlebot.
• Answer Engine Optimization (AEO) — The practice of optimizing content, schema, and brand signals so that AI answer engines like ChatGPT, Perplexity, and Gemini cite your brand accurately when answering user queries.