robots.txt tester
Enter a site and the paths you care about. We fetch its robots.txt and test each path the way Google does (the most specific user-agent group, the longest rule, Allow winning a tie, * and $ wildcards), for Googlebot, Bingbot and the AI crawlers.
Free, no sign-up. Fetched from AskYourStack in Germany as the server sends the page (no JavaScript).
How Google reads robots.txt
A crawler picks one group: the one whose User-agent line names it most specifically, or the * group when none does. Inside that group the longest matching rule wins, and when an Allow and a Disallow match with the same length, Allow wins. * matches any run of characters and $ anchors the end of the address.
Rules are about crawling, not indexing: a page blocked in robots.txt can still appear in Google with no description if other sites link to it. To keep a page out of the index, let it be crawled and use a noindex tag instead.
AI crawlers have their own names
GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot and Google-Extended (Google's AI training, not Search) each read robots.txt under their own name. Blocking them does not change how Google Search crawls your site; blocking Googlebot does.
What usually goes wrong
A Disallow: / left over from a staging site, CSS and JavaScript folders blocked so Google cannot render the page, a search or filter rule written without a wildcard that never matches, and a missing Sitemap line.
Ask your AI instead
AskYourStack connects Claude, ChatGPT or Cursor to your Search Console, Google Analytics and the same page tools. Ask in plain words and it checks, compares and explains:
“Check robots.txt on my site and tell me if anything important is blocked for Google or AI crawlers.”
Start free Free plan: one Search Console site, no card.Questions
Is this the same as Google's robots.txt report?
It follows the same matching rules Google documents, and tests any site, not only yours. Search Console's report shows what Google itself fetched and when.
Do you store the sites I test?
We keep the address and the time of each test for 30 days to limit abuse. Nothing else.
How many tests can I run?
Twelve an hour from one address. With a free AskYourStack account your AI can run them whenever you ask.