How to use Robots.txt Tester
Test path rules for a chosen user-agent using longest-match precedence. Local rule evaluation uses user-agent matching and longest path match, with Allow winning ties. It does not crawl the site, enforce access controls or guarantee how every bot behaves.
- Set Input.
- Set User-agent.
- Set Path.
- Run the tool, review the result and its stated limitations, then copy or download if needed.
Example & practical use
Test path rules for a chosen user-agent using longest-match precedence. The example below demonstrates Input; User-agent; Path.
Example input:
{
"text": "User-agent: *\nDisallow: /private\nAllow: /private/public",
"agent": "Googlebot",
"path": "/private/public"
}
Expected result (relevant fields):
{
"allowed": true
}Features & result limitations
Local rule evaluation uses user-agent matching and longest path match, with Allow winning ties. It does not crawl the site, enforce access controls or guarantee how every bot behaves.
Frequently asked questions
What inputs does Robots.txt Tester accept?
Input; User-agent; Path
How should I interpret the Robots.txt Tester result?
Local rule evaluation uses user-agent matching and longest path match, with Allow winning ties. It does not crawl the site, enforce access controls or guarantee how every bot behaves.
Where does Robots.txt Tester process my data?
Inputs are processed locally in your browser. Submitted text is not sent to external services. A successful-run event increments an aggregate tool counter without your input.