A robots.txt file is four or five lines of plain text, and one wrong character in it decides whether Google crawls your product pages or skips them for weeks. I have seen a single stray slash hide an entire blog from search, and a misspelled directive leave a staging folder wide open. Those mistakes stay invisible until traffic drops.
That is why I test every robots.txt change before it goes live, and the good news is that the best robots.txt testing tools are free. Google retired its old standalone tester, so the job now sits with a mix of webmaster consoles, browser based checkers, desktop crawlers and open source parsers.
For this guide I ran the same test file through the free robots.txt testers worth your time, and I added a small experiment of my own: two parsers, six edge cases and 100,000 URL checks. One result surprised me, and it changes which robots.txt checker you should trust for typos.
Below you get what a robots.txt tester checks, my test results, the 10 free tools with what each does best, a comparison table, a step by step testing routine and answers to common questions.
- What a Free Robots.txt Testing Tool Checks
- My Robots.txt Test: Same File, Two Parsers
- The 10 Best Free Robots.txt Testing Tools
- 1. Google Search Console Robots.txt Report
- 2. Bing Webmaster Tools Robots.txt Tester
- 3. Tame the Bots Robots.txt Testing Tool
- 4. Spotibo Robots.txt Tester
- 5. Screaming Frog SEO Spider
- 6. Nuxt SEO Robots.txt Validator
- 7. Growtika Robots.txt Checker
- 8. Ralf van Veen Robots.txt Validator
- 9. Google Robotstxt Open Source Library
- 10. Python Robotparser and Protego
- Robots.txt Testing Tools Compared
- How to Test Robots.txt Step by Step
- Robots.txt Mistakes a Free Tester Catches
- Free Robots.txt Testing Tools FAQs
What a Free Robots.txt Testing Tool Checks
A good robots.txt testing tool answers three questions:
- Is the file valid? Syntax checks catch rules with zero User-agent line above them, broken wildcards, relative sitemap URLs and unknown directives.
- Is this URL allowed or blocked? URL tests match a path against the rules for a chosen crawler and show the exact rule that wins.
- What does each bot see? Bot simulation shows the result for Googlebot, Bingbot, GPTBot, ClaudeBot and other crawlers, since each one reads its own group of rules.
The second question matters most. Google follows the most specific matching rule (the longest path), and when an Allow and a Disallow rule tie, the less restrictive Allow wins. A robots.txt checker built on the same logic gives you the answer Googlebot gives; a checker built on simple first match logic gives you a guess.
My Robots.txt Test: Same File, Two Parsers
I wrote a test file with the patterns that cause most real problems:
User-agent: *
Disallow: /shop/
Allow: /shop/sale/
Disallow: /*?sort=
Disallow: /*.pdf$
User-agent: GPTBot
Disallow: /Then I ran it, plus six edge cases, through Python 3.13 robotparser and Protego, an open source parser that follows the Google specification. Here is what came back for Googlebot:
Robots.txt Test Results for Googlebot
| URL or Case | Python robotparser | Protego (Google style) |
|---|---|---|
| /shop/ | Blocked | Blocked |
| /shop/sale/shoes | Allowed | Allowed |
| /blog/post?sort=asc | Blocked | Blocked |
| /files/guide.pdf | Blocked | Blocked |
| /files/guide.pdf?v=2 | Allowed | Allowed |
| Tie: Allow /p and Disallow /p | Allowed | Allowed |
| Rule order: Disallow /shop above Allow /shop/sale | Allowed | Allowed |
| Path case: Disallow /Private, URL /private | Allowed | Allowed |
| Typo: Dissallow /private | Allowed | Blocked |
Three takeaways from this robots.txt test:
- The $ anchor is strict. /files/guide.pdf is blocked, yet /files/guide.pdf?v=2 is allowed because the URL ends with v=2 and the rule expects .pdf at the very end. Add a second rule such as Disallow: /*.pdf? if you want both blocked.
- Paths are case sensitive. Disallow: /Private leaves /private open.
- Typos split the tools. The official Google parser forgives common misspellings such as Dissallow and treats them as Disallow. Python ignores the line. The same file gives opposite answers depending on the robots.txt tester you pick, so test typos with a tool built on the Google library and fix the spelling either way.
Speed was a non issue: 100,000 URL checks took 1.95 seconds in Python robotparser and 1.28 seconds in Protego, about 51,000 and 78,000 checks per second. Bulk testing a full URL list against robots.txt is fast and free.
Every GPTBot URL came back blocked in both parsers, which confirms that a bot with its own group ignores the rules under User-agent: *. Our bot traffic statistics show why that separation matters as AI crawlers grow.
The 10 Best Free Robots.txt Testing Tools
1. Google Search Console Robots.txt Report
Google Search Console: The robots.txt report under Settings in Search Console shows the robots.txt files Google found for the top 20 hosts on your property, when Google last crawled each one, and any errors or warnings. It also has a request recrawl button for emergencies, for example after you remove an accidental Disallow: / from a live site.
Pair it with the URL Inspection tool to confirm whether a specific page is blocked by robots.txt. The report shows the version Google holds, which is the truth that counts, while a URL level allow or block test with custom rules needs one of the tools below.
| Cost | Free with a verified property |
| Tests draft rules | Live data only |
| Best for | Seeing the robots.txt version Google actually uses |
2. Bing Webmaster Tools Robots.txt Tester
Bing Webmaster Tools: Bing has the most complete robots.txt tester inside a search engine console. It loads your file in an editor across all four variants (http, https, www and bare domain), lets you edit the rules, and tests any URL against them as Bingbot or BingAdsBot.
Fetch Latest pulls the newest live version, and once you are happy with an edit you download the file, upload it to your root folder and ask Bing to refresh its copy. Bing results also feed Copilot and other AI search products, so this check covers more than classic Bing search.
| Cost | Free with a verified site |
| Tests draft rules | Yes |
| Best for | Editing and testing in one place |
3. Tame the Bots Robots.txt Testing Tool
Tame the Bots: This is my first choice for URL level testing. It runs on the Google open source robots.txt parser, accepts a list of URLs across several domains, and shows which rule allowed or blocked each one. A custom mode tests draft rules before you publish.
The user agent list is the most complete I found: Googlebot and its News, Image and Video variants, AdsBot, StoreBot, GoogleOther, Google-Extended, Bingbot, Applebot, GPTBot, OAI-SearchBot and ChatGPT-User, plus a custom field. It also copies the way Googlebot-Image falls back to Googlebot rules. A Chrome extension and a beta MCP server for AI assistants round it out.
| Cost | Free (donation supported) |
| Tests draft rules | Yes |
| Best for | Google accurate URL lists and AI bot checks |
4. Spotibo Robots.txt Tester
Spotibo: Spotibo also uses the same open source library Google relies on, so its answers match Googlebot. It handles encoded and special characters such as commas and accented letters correctly, which trips up simpler checkers on multilingual sites.
It reflects Google quirks too, for example AdsBot-Google ignoring the global User-agent: * group. The extra I like is free robots.txt change alerts: create a free account and you get an email alert every time the file changes.
| Cost | Free |
| Tests draft rules | Yes |
| Best for | Multilingual URLs and change alerts |
5. Screaming Frog SEO Spider
Screaming Frog SEO Spider: The desktop crawler tests robots.txt at site scale. The free version crawls up to 500 URLs and includes the custom robots.txt feature, where you paste draft rules and see which crawled URLs they block before anything goes live.
Crawl reports flag every URL blocked by robots.txt along with the matching line, which is the fastest way to spot an important section caught by a broad rule. The paid licence costs $279 a year and removes the URL cap.
| Cost | Free up to 500 URLs |
| Tests draft rules | Yes |
| Best for | Whole site impact of a rule change |
6. Nuxt SEO Robots.txt Validator
Nuxt SEO Robots.txt Validator: This free validator takes a pasted file or fetches a live one, reports syntax issues and the matched rule, and tests one user agent against one path. It catches rules missing a User-agent line, relative Sitemap URLs, blocked asset folders, case problems and conflicting Allow and Disallow pairs.
It also reads the newer AI content signals, the IETF Content-Usage draft and Cloudflare Content-Signal, which few robots.txt checkers report today.
| Cost | Free |
| Tests draft rules | Yes |
| Best for | Syntax checks and AI content signals |
7. Growtika Robots.txt Checker
Growtika Robots.txt Checker: Growtika focuses on AI crawler access. Paste your file, run the audit and you get a total score, a syntax error list and a status card for 16 crawlers across AI, search and social, including GPTBot, ClaudeBot, PerplexityBot, Googlebot and Google-Extended.
Each issue comes with a fix snippet you copy into your file. It works well for a quick AI visibility check; for Google exact URL answers, confirm with Tame the Bots or Spotibo.
| Cost | Free, zero sign up |
| Tests draft rules | Yes (paste) |
| Best for | AI crawler access audit |
8. Ralf van Veen Robots.txt Validator
Ralf van Veen Robots.txt Validator: Enter a domain and this validator reviews the live file for syntax and format errors, duplicate or conflicting rules, misused wildcards, unknown user agents and the sitemap location. It warns when important sections look blocked.
It works as a file health check for the live robots.txt, and it makes a good second opinion after you edit a long robots.txt file.
| Cost | Free |
| Tests draft rules | Live file only |
| Best for | Quick health check of a live file |
9. Google Robotstxt Open Source Library
Google robotstxt library: This is the C++ parser Google uses in production, released as open source. Developers build it and test a robots.txt file against a URL and user agent from the command line, getting the same answer Googlebot gets.
It fits engineering teams who want robots.txt tests in a build pipeline, so a bad rule fails the deploy before it reaches production.
| Cost | Free, open source |
| Tests draft rules | Yes |
| Best for | Automated tests in CI pipelines |
10. Python Robotparser and Protego
Protego: These are the two parsers from my experiment. Python ships urllib.robotparser in the standard library, and Protego is the Google style parser behind Scrapy. A ten line script tests thousands of URLs from a crawl export in seconds.
Use Protego when you want Google behaviour, including typo tolerance. In my run it handled about 78,000 checks per second on a laptop class container.
| Cost | Free, open source |
| Tests draft rules | Yes |
| Best for | Bulk URL lists from crawls or logs |
Robots.txt Testing Tools Compared
Free Robots.txt Testers at a Glance
| Tool | Type | Google Parser | AI Bots | Draft Rules | Sign Up |
|---|---|---|---|---|---|
| Search Console report | Console | Yes (live data) | Google only | Live only | Verified site |
| Bing Webmaster Tools | Console | Bing logic | Bingbot | Yes | Verified site |
| Tame the Bots | Web | Yes | Yes | Yes | Free |
| Spotibo | Web | Yes | Google bots | Yes | Free |
| Screaming Frog | Desktop | Google style | Custom UA | Yes | Free to 500 URLs |
| Nuxt SEO | Web | Rule match | Signals | Yes | Free |
| Growtika | Web | Audit | 16 bots | Yes | Free |
| Ralf van Veen | Web | Audit | Flags unknown | Live only | Free |
| Google robotstxt | Code | Yes | Any | Yes | Free |
| Protego and robotparser | Code | Protego: Yes | Any | Yes | Free |
My setup: Tame the Bots for URL checks, Screaming Frog for site wide impact, Bing Webmaster Tools for edits, and the Search Console robots.txt report to confirm what Google holds. Add Growtika when AI crawler access is the question.
How to Test Robots.txt Step by Step
Step 1: Copy the live file
Open yourdomain.com/robots.txt and save the current version so you have a rollback.
Step 2: Write the change as a draft
Edit the copy, keeping one group per crawler and the most specific paths you need.
Step 3: Test key URLs in a free tester
Paste the draft into Tame the Bots or Spotibo and test your homepage, a category, a product, a blog post, a CSS or JS file and every URL the new rule targets.
Step 4: Check AI crawlers
Run the same URLs as GPTBot, ClaudeBot, OAI-SearchBot and Google-Extended to confirm AI access matches your policy. Our guide to GEO basics covers why that choice affects AI visibility.
Step 5: Crawl the site with the draft
Use Screaming Frog custom robots.txt to see every blocked URL across up to 500 pages for free.
Step 6: Publish and confirm
Upload the file, check the robots.txt report in Search Console and request a recrawl if the change is urgent.
Step 7: Watch for changes
Turn on Spotibo alerts or use one of these website change tracking tools so an accidental edit by a plugin or developer gets caught the same day.
Robots.txt Mistakes a Free Tester Catches
Common Robots.txt Errors and Fixes
| Mistake | What Happens | Fix |
|---|---|---|
| Disallow: / left from staging | Whole site blocked | Remove it and request a recrawl |
| Blocking /wp-content/ or JS and CSS folders | Google renders pages badly | Allow asset folders |
| $ anchor on file types | Versions with parameters stay crawlable | Add a second rule with ? |
| Wrong case in paths | Rule misses the real URL | Match the exact case |
| Rules above any User-agent line | Rules ignored | Start every group with User-agent |
| Relative Sitemap URL | Sitemap ignored | Use the full https URL |
| Using Disallow to remove pages | Page stays indexed from links | Use noindex and allow crawling |
| Misspelled directives | Different tools give different answers | Correct the spelling |
Remember that robots.txt controls crawling. Indexing is a separate step. A blocked URL still shows in results if other sites link to it, so a page you want out of Google needs a noindex tag and an open path for Googlebot to read it.
Free Robots.txt Testing Tools FAQs
Does Google Still Have a Robots.txt Tester?
Google replaced the old standalone tester with the robots.txt report in Search Console. The report shows the fetched files, crawl dates and errors for your top 20 hosts. For URL level testing of draft rules, use Tame the Bots or Spotibo, both built on the Google open source parser.
What Is the Best Free Robots.txt Tester?
Tame the Bots is my top pick for accurate URL tests across Google and AI crawlers. Bing Webmaster Tools is the best free option for editing and testing in one place.
How Do I Test Robots.txt for AI Crawlers?
Run your URLs as GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot and Google-Extended in Tame the Bots or Growtika. Each AI bot follows its own User-agent group when one exists.
How Long Does Google Take to See a Robots.txt Change?
Google caches robots.txt for up to 24 hours. Use the request recrawl option in the Search Console robots.txt report for urgent fixes.
Does Google Ignore Crawl-delay?
Yes. Googlebot ignores Crawl-delay, while Bing and some other crawlers respect it.
Is Robots.txt Testing Possible for Thousands of URLs?
Yes. Screaming Frog tests 500 URLs free, and a short Protego script checked 100,000 URLs in 1.28 seconds in my test.
Find more tools:
