Robots.txt Generator Online — Free, With AI Presets
Build a correct robots.txt from simple rules — user-agent groups, allow/disallow paths, crawl-delay, sitemap location — or start from a preset for common setups.
- Disallow rules are prefix matches: /api blocks /api and everything under it.
- Sitemap lines must be absolute URLs and are ignored by some older crawlers.
- Robots.txt is a request, not a lock — anything under it is still reachable by URL.
Every conversion in this toolkit runs in your browser with the standard JS APIs. Your input is never uploaded, stored, or logged.
The mental model: groups and prefixes
A robots.txt is a sequence of groups: each User-agent line starts a group, and every Allow/Disallow/Crawl-delay in that group applies to matching crawlers. A crawler uses the group whose name matches it exactly, or the * group when none does. The most specific matching group wins — rules from other groups are not merged in.
Paths are prefix strings, not globs. Disallow: /admin blocks /admin, /admin/login and /administrator alike. The Allow directive resolves conflicts: Google and Bing pick the most specific (longest) matching rule, so Allow: /admin/login inside a group that disallows /admin opens exactly that path.
Presets and honesty about limits
The presets cover the requests that come up weekly: fully open, fully closed (staging), block AI crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended) while leaving search engines alone, and a standard site that hides /api and /admin. Edit the generated text freely — it is plain text by design.
Two things robots.txt cannot do: hide content (disallowed URLs can still be linked and indexed without crawling) and enforce anything (it is advisory). For real privacy you need authentication — for de-indexing, meta robots or X-Robots-Tag headers.
Rolling it out without hurting your own site
The failure mode worth planning for is blocking something a crawler needs. A Disallow on a directory holding your CSS or JavaScript stops Google from rendering pages, which hides the content quality it would otherwise assess. A whole-site Disallow copied from staging into production is the one that quietly costs traffic for weeks before anyone notices.
So verify after deploying rather than before: fetch a live URL with a crawler tester and confirm the rules resolve the way you intended, keep the staging block somewhere that cannot be promoted, and treat every path as public anyway — robots.txt is a request to well-behaved crawlers, not an access rule.
Frequently asked questions
Where does robots.txt go?
Exactly one place: the site root — https://example.com/robots.txt. Subdirectory copies are ignored by compliant crawlers.
How do I block AI crawlers but keep Google?
Add separate groups for GPTBot, ClaudeBot, CCBot, and Google-Extended with Disallow: /, then a User-agent: * group that allows the site. The preset labeled 'Block AI crawlers' generates exactly that.
Does blocking in robots.txt remove pages from Google?
No — it prevents crawling, not indexing. A blocked URL can still appear in results if other pages link to it. To de-index, use a noindex meta tag or X-Robots-Tag header on a crawlable page.