Skip to content
🤖

Robots TXT Checker

SEO

Edit robots.txt, inspect rule groups and test a path against crawler rules.

0
0
Input

Validate access rules and test a path for a crawler product token. robots.txt is not access control. Empty Disallow permits crawling; crawler extensions are reported separately.

Paste robots.txt or fetch a URL that permits cross-origin requests. Edit and validate directives, inspect user-agent groups and test paths.

See more about the Robots TXT Checker

Matching combines applicable groups, supports wildcards and end anchors, and prefers Allow on equally specific matches. Empty Allow and Disallow rules have no effect. Extension warnings do not mean every crawler behaves identically; robots.txt is not access control.

Key features

  • Network access required for URL requests

Related Blog Posts

Quick answers for Robots TXT Checker

Can I paste or fetch robots.txt?
Yes. Paste and edit the rules locally, or fetch a URL that permits CORS. Validation reports errors and extension warnings; it does not enforce access restrictions.
How does path testing work?
Matching user-agent groups are combined. The most specific matching rule wins, with Allow winning ties. Wildcards, end anchors and empty rules are handled; empty Allow or Disallow values have no effect.
Does this guarantee how every crawler behaves?
No. Crawler extensions and implementation differences still need review. robots.txt is not authentication or access control.