robots.txt
robots.txt is a text file at the root of a website that tells crawlers which paths they may or may not fetch.
Updated June 1, 2026
In depth
For AI citations, robots.txt now doubles as the AI crawler allowlist. Explicit Allow directives for GPTBot, PerplexityBot, ClaudeBot, and Google-Extended remove ambiguity and make your intent auditable.
Related terms