Robots.txt Checker Online

Quickly test robots.txt rules against example URLs

If no User-agent set in rules, we will auto assume `User-agent: *`


Useful resources for robots.txt files: