Configuration
What is robots.txt?
The robots.txt file tells indexing robots (Google, Bing, but also AI crawlers like GPTBot and ClaudeBot) which pages of your site they can or cannot crawl.
Why is it important?
- Control robot access to your sensitive pages
- Optimize crawl budget for SEO
- Allow or block AI crawlers (GEO)