robots.txt
File that tells crawlers (Google, GPTBot, ClaudeBot) what they may fetch.
Why this matters
Many sites accidentally block ChatGPT or Perplexity here — and lose every AI citation.
File that tells crawlers (Google, GPTBot, ClaudeBot) what they may fetch.
Many sites accidentally block ChatGPT or Perplexity here — and lose every AI citation.