robots.txt Generator
Create robots.txt correctly
The robots.txt in the root directory controls which parts of your website search engines may crawl. The generator creates a valid file with the common directives.
- Block paths: Enter directories that should not be indexed (e.g.
/wp-admin/). - Sitemap: The reference to the XML sitemap helps search engines find all pages.
- Important: robots.txt is a recommendation, not access protection – secure sensitive areas separately.