robots.txt Generator
Create a robots.txt file for your website for free. Add user-agent groups with allow and disallow rules, set a crawl-delay and link your sitemap, then copy or download the file. Everything runs in your browser, with nothing sent to servers.
One path per line, for example /admin
User-agent: * Disallow:
How to use
- 1Set the user-agent and the paths to allow or disallow.
- 2Add more groups or your sitemap URL if needed.
- 3Copy or download the generated robots.txt.
What it's for
- Block crawlers from admin or private folders.
- Point search engines to your sitemap.
- Set different rules for different bots.
- Create a starter robots.txt for a new site.
Frequently asked questions
Is the tool free?
Yes. It's completely free with no usage limits.
Where do I put the robots.txt file?
Upload it to the root of your domain so it's reachable at https://yourdomain.com/robots.txt. That's the only place crawlers look for it.
What does an empty Disallow mean?
A blank 'Disallow:' line allows everything for that user-agent. To block a path, add it after Disallow, like '/admin'.
Does robots.txt guarantee a page stays private?
No. It asks well-behaved crawlers not to crawl a path, but it doesn't hide or secure the page. Use authentication or noindex for truly private content.
Do I need an account?
No. The tool works right away, with no signup or login.