robots.txt Generator

✓ Link copied

Create a correct robots.txt file for your site without memorizing the syntax — entirely in your browser. Add one or more crawler groups, each targeting a User-agent like * or Googlebot, then list the paths to Allow and Disallow, set an optional Crawl-delay, and add your Sitemap URLs. The generated file updates live as you type, with groups separated cleanly and sitemap directives placed at the end where crawlers expect them. Copy the result in one click or download it as robots.txt to drop into your site's root. Free, fast, and completely private — nothing is uploaded or stored.

Group 1

Generated robots.txt

How to use

For each crawler rule, enter the User-agent it applies to (use * for all crawlers, or names like Googlebot and Bingbot separated by commas). List the paths to Allow and Disallow, one per line, and optionally set a Crawl-delay in seconds. Click Add group to target specific crawlers with different rules. Finally, add your Sitemap URLs, one per line. The robots.txt on the right rebuilds instantly — when it looks right, click Copy or Download and place the file at the root of your domain (https://example.com/robots.txt).

Frequently asked questions

Where do I put the robots.txt file?
It must live at the root of your domain — for example https://example.com/robots.txt. Crawlers only look there; a robots.txt placed in a subdirectory is ignored. Each subdomain (such as blog.example.com) needs its own robots.txt.
Does Disallow guarantee a page stays out of search results?
No. Disallow tells well-behaved crawlers not to fetch a path, but a disallowed URL can still appear in search results if other pages link to it. To reliably keep a page out of an index, allow crawling and use a noindex meta tag, or protect the page behind authentication. robots.txt is a crawling directive, not an access-control or privacy mechanism.
Is my input sent anywhere?
No. The robots.txt is assembled entirely in your browser as you type. Your user agents, paths, crawl delays, and sitemap URLs are never transmitted, logged, or stored on any server, and the tool works without an account.