Web Tools tools.bryantsmith.com

SEO & Performance

Robots.txt Tester

Paste robots.txt rules, pick a user-agent, and instantly see whether a URL is allowed or blocked — with the exact rule that decides it. Your data never leaves the browser.

Runs in your browser v1.0.0

Test request

updates live

Crawler name, or * for all unnamed crawlers.

A full URL or a path beginning with /.

robots.txt

paste or type

Rules are evaluated in your browser using the Robots Exclusion Protocol: longest matching user-agent and longest matching path win, with Allow breaking ties.

Result

0 rules

How it works

The pasted file is parsed into user-agent groups. The group is chosen by the most specific matching crawler name, and the winning path rule is the longest pattern that matches your URL (* matches any characters and a trailing $ anchors the end). When two rules are equally specific, Allow wins. Everything runs locally, so confidential staging rules stay off the network.

Related tools

  • Robots.txt GeneratorBuild a valid robots.txt with allow/disallow rules, crawl-delay and sitemap — with a live preview in your browser.
  • Social Preview TesterPreview Open Graph and Twitter Card metadata live, with length checks and copy-ready meta tags.
  • Canonical URL CheckerCompare a page URL with its canonical URL and flag common canonicalization problems, entirely in your browser.
  • JSON-LD Validator HelperValidate JSON-LD syntax and flag common structural omissions in your structured data — entirely in your browser.
  • Schema.org GeneratorBuild common schema.org JSON-LD structured data with guided fields and copy it straight into your pages.
  • SERP Snippet PreviewPreview your title, description, and URL as a Google-style search snippet at desktop and mobile widths.

More SEO & Performance tools