Loading Legacy
Create a search-engine friendly robots.txt file for your site.
Generated Content
Robots.txt Generator helps set crawler rules and sitemap structure so important pages are discovered and indexed efficiently.
Generate a custom robots.txt file to control how search engines crawl your website.
Select allowed/disallowed paths.
Add your sitemap URL.
Download or copy the robots.txt content.
Rules and sitemap output are generated from your configuration and should be reviewed to avoid blocking important URLs.
Place it in the root directory of your website.
It is highly recommended for better SEO control.