Robots.txt Tester

robots.txt Tester compared with Google Search Console and Screaming Frog
robots.txt Tester	Google Search Console	Screaming Frog
Function: This is an online tool from Rank Math that tests and validates a robots.txt file. It is meant to find errors and let you review the directives.	Function: The robots.txt report under Settings lists the files Google found for the top 20 hosts of a Domain property. It shows fetch status, check time, file size, a count of parsing issues and the last fetched contents with errors and warnings highlighted.	Function: This is a desktop crawler. Under Configuration, Robots.txt you can respect the file, ignore it, or ignore it but report its status. URLs blocked by a rule appear with status code 0 and the status Blocked by Robots.txt.
	Scope and limits: The report is available only for domain-level properties, not for URL-prefix properties with a path. To check whether one URL is blocked, the URL Inspection tool is used instead.	Scope and limits: The crawl follows the user agent set in the configuration, with presets such as Googlebot and Bingbot or a custom string. A custom robots.txt can be downloaded, edited and tested inside the tool, and it overrides the live file for that crawl only.
	Changes and corrections: The report offers a way to request a recrawl of a robots.txt file for emergency situations. It shows fetch history for the last 30 days.	Changes and corrections: Edits made in the custom file are not written to the server. They must be copied to the live site by hand.
Cost and platform: The tool is described as free on the Rank Math page. It runs online.		Cost and platform: The free version has a crawl limit of 500 URLs. A paid licence is sold per seat on an annual basis. Builds exist for Windows, macOS and Linux.

robots.txt Tester compared with Google Search Console and Screaming Frog

robots.txt Tester

Google Search Console

Screaming Frog

Function: This is an online tool from Rank Math that tests and validates a robots.txt file. It is meant to find errors and let you review the directives.

Function: The robots.txt report under Settings lists the files Google found for the top 20 hosts of a Domain property. It shows fetch status, check time, file size, a count of parsing issues and the last fetched contents with errors and warnings highlighted.

Function: This is a desktop crawler. Under Configuration, Robots.txt you can respect the file, ignore it, or ignore it but report its status. URLs blocked by a rule appear with status code 0 and the status Blocked by Robots.txt.

Scope and limits: The report is available only for domain-level properties, not for URL-prefix properties with a path. To check whether one URL is blocked, the URL Inspection tool is used instead.

Scope and limits: The crawl follows the user agent set in the configuration, with presets such as Googlebot and Bingbot or a custom string. A custom robots.txt can be downloaded, edited and tested inside the tool, and it overrides the live file for that crawl only.

Changes and corrections: The report offers a way to request a recrawl of a robots.txt file for emergency situations. It shows fetch history for the last 30 days.

Changes and corrections: Edits made in the custom file are not written to the server. They must be copied to the live site by hand.

Cost and platform: The tool is described as free on the Rank Math page. It runs online.

Cost and platform: The free version has a crawl limit of 500 URLs. A paid licence is sold per seat on an annual basis. Builds exist for Windows, macOS and Linux.