How to use the Robots.txt Generator
- Choose the user-agent.
- Add disallow paths.
- Add your sitemap URL.
- Copy or download robots.txt.
Frequently Asked Questions
Does this block Google automatically?
Only after you publish the generated file on your domain.
Can robots.txt guarantee deindexing?
No. Use appropriate meta robots or Search Console controls as applicable.
Writing a robots.txt file
Enter the paths you want crawlers to avoid, such as admin areas or internal search pages, and add your sitemap address. Save the output as robots.txt in the root of your site so it is reachable at yourdomain.com/robots.txt.
Be careful: blocking a path does not remove it from search results if other pages link to it, and robots.txt is publicly readable, so never use it to hide private content. Do not block CSS or JavaScript files that pages need to render. A mistaken rule such as blocking the whole site can remove it from search, so double-check before uploading.
Limits: the generator builds simple rules for one user-agent. Complex sites may need separate rules for different crawlers.