Robots.txt Generator
Build a correct robots.txt in seconds — block paths, block AI crawlers, add your sitemap. Copy or download.
Blocked paths
Your robots.txt
User-agent: * Disallow: /admin/
Upload this file to your site's root so it's reachable at yourdomain.com/robots.txt.
About this tool
robots.txt is a plain-text file at your site's root that tells crawlers which parts of the site they may visit. It's the first thing a well-behaved bot requests, and a one-character mistake in it can accidentally deindex an entire site — "Disallow: /" blocks everything, "Disallow:" blocks nothing, and that difference has ruined more than a few launch days. A generator exists precisely so nobody hand-types that line.
Start from your intent — allow all crawling (the right default for almost every public site) or block everything (for staging and pre-launch sites) — then add specific paths to keep crawlers out of admin panels, checkout flows, internal search results, or anything else that shouldn't appear in search. The sitemap line is a one-field addition and worth including: it's how crawlers discover your sitemap without you submitting it manually to each search engine.
The modern option most generators lack: a one-click block for AI training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot) for site owners who want search indexing but not AI training use. One honest caveat, stated plainly: robots.txt is a convention, not a lock — reputable crawlers obey it; malicious ones ignore it. It's traffic direction, not security. Copy the result or download the file, then upload it to your site root.
How to use the Robots.txt Generator
- 1Choose your base policy: allow all crawling, or block everything (for staging sites).
- 2Add any paths to block — /admin/, /checkout/, internal search pages.
- 3Optionally block AI training crawlers and add your sitemap URL.
- 4Download robots.txt and upload it to your site's root directory.
Frequently asked questions
Where does robots.txt go?
At the root of your domain, reachable at yourdomain.com/robots.txt — nowhere else works. Crawlers only look at that exact address. On most hosting, that means the top-level public folder; on platforms like Next.js or WordPress, there's usually a built-in way to serve it.
Does robots.txt actually block bad bots?
No — it's a voluntary convention. Search engines and reputable crawlers respect it; scrapers and malicious bots ignore it completely. Use it to guide legitimate crawling, not as security. Anything truly private needs authentication, not a robots rule.
Should I block AI crawlers?
It's a genuine trade-off. Blocking GPTBot, ClaudeBot and similar prevents your content being used for AI training, but may also reduce your visibility in AI-powered search and assistants, which increasingly send real traffic. There's no universal right answer — this tool makes either choice one click.
Will 'Disallow' remove a page from Google?
Not reliably — it stops crawling, but the URL can still appear in results if other sites link to it. To keep a page out of search, use a noindex meta tag (and let it be crawled so the tag is seen), or password-protect it.
Related tools
Meta Tag Generator
Generate correct HTML meta tags — with live character counters for title and description limits.
Open toolOpen Graph Generator
Generate og: and Twitter card tags — with a live preview of how your link will look when shared.
Open toolSlug Generator
Turn any title into a clean, SEO-friendly URL slug — live as you type, accents and symbols handled.
Open toolUTM Link Builder
Build campaign tracking URLs with UTM parameters — presets for the common channels, properly encoded.
Open tool