Robots.txt Generator

Build a robots.txt without touching syntax.

Free robots.txt generator. Add allow/disallow rules for any user-agent, attach your sitemap URL, then copy or download a valid robots.txt file.

Loading tool…

If the tool stays on this message, make sure JavaScript is enabled, then reload the page.

About Robots.txt Generator

The robots.txt file is a small text file that lives at the root of your website and tells search-engine crawlers which parts of the site they may and may not crawl. It's the first thing a well-behaved bot reads when it arrives. This generator builds a valid file visually. Set a user-agent (use * for all crawlers), add Allow and Disallow rules, attach your sitemap URL, then copy or download the finished file.

The most important thing to understand is what robots.txt is not. It is not access control: it's a polite request, and malicious bots ignore it cheerfully. Never use it to "protect" private content, anything you disallow is still publicly reachable by URL. It also doesn't remove pages from Google's index; a disallowed page can still appear in search results. For that, you need noindex directives on the pages themselves.

The classic disaster is a stray "Disallow: /" that blocks the entire site from crawling, always double-check your rules before uploading, and place the file exactly at the domain root (example.com/robots.txt); it works nowhere else. Include the Sitemap: line pointing at your sitemap so crawlers discover your pages efficiently. Once your rules are live, verify them in Google Search Console.

How to use Robots.txt Generator

  1. Set the user-agent (use * for all crawlers).
  2. Add Allow/Disallow rules with paths starting in /.
  3. Add your sitemap URL, generate, then download.

Why use this tool?

  • Visual rule builder, no syntax to memorize
  • Sitemap line included automatically
  • Copy or download the finished file

Tips for best results

  • Upload the file to your domain root, example.com/robots.txt. It works nowhere else.
  • Use * as the user-agent unless you genuinely need bot-specific rules.
  • Disallow is a polite request, not security, never rely on it to hide private content.
  • Double-check you didn't write "Disallow: /", blocking your whole site is the classic outage.
  • Include the Sitemap: line so crawlers find your pages efficiently.
  • To remove a page from Google, use noindex on the page itself, robots.txt won't do it.

Frequently asked questions

Where do I put the file?
Upload it to your site's root, e.g. example.com/robots.txt. Crawlers only look there, a robots.txt in a subdirectory is ignored.
Does robots.txt block bad bots?
No. It's a polite request that well-behaved crawlers follow; malicious bots ignore it. It's not access control.
Is anything uploaded?
No. The file is built in your browser; nothing leaves your device.
Will Disallow remove a page from Google?
No, a disallowed page can still appear in results. To remove a page from the index, use a noindex directive on the page itself.

Related tools

FAQ Schema GeneratorBuild valid FAQPage JSON-LD in seconds. SERP Snippet PreviewSee your Google snippet before you publish. Meta Title CheckerCheck title length in characters and pixels. URL Slug GeneratorTurn any title into a clean URL slug.