What Is Robots.txt and Why It Matters for SEO
Robots.txt is a plain text file placed at the root of your website that communicates with search engine crawlers. Developed as part of the Robots Exclusion Protocol in 1994, it gives website owners a way to manage how automated bots access their content. From an SEO perspective, robots.txt is one of the most powerful and most misused tools available. Used correctly, it helps you focus your crawl budget on high-value pages. Used incorrectly, it can wipe your site from search results overnight.